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Executive Summary 


Introduction 


The dramatic evolution of computer and communications technology, coupled with an ever-increasing 
emphasis on cost-effectiveness, underscores the potential of technology-based learning (TBL), or e- 
learning, in the public workforce investment system. The U.S. Department of Labor (the Department), 
Employment and Training Administration (ETA) launched a national initiative in 2008 to systematically 
explore TBL’s role and its adoption within the workforce system. To date, several demonstration grants 
and pilots funded under the TBL Initiative have examined the structure and outcomes of specific training 
programs. Despite these insights, comparatively little was known about the emphasis on TBL at the state 
policy level or its systematic adoption and use at the local level by American Job Centers (AJCs) and 
other providers. 


With funding from the Department, Abt Associates conducted a study during 2012-2013 to collect 
descriptive information about the use of TBL at the state and local levels of the workforce system. Data 
about state policies and support for TBL were collected in an online survey of state workforce agency 
(SWA) administrators (82 percent response rate), and data about local implementation of TBL were 
collected in an online survey of the Executive Directors of Local Workforce Investment Boards (LWIBs; 
69 percent response rate). This report presents the findings from those surveys. Specifically, it provides a 
description of SWAs’ efforts to promote and support TBL, as well as actual adoption and use of TBL at 
the local level in providing Workforce Investment Act (WIA) services. These include WIA Title I core, 
intensive, and training services and WIA Title II educational services. The report also identifies factors 
that appear to facilitate or impede state supports for TBL and local use of TBL. 


For purposes of this research, TBL is defined as an umbrella term for any learning via electronic 
technology. Further, TBL is subdivided into two distinct uses of electronic technology. The first is the 
use of technology to promote skills building that supports academic, occupational training, or personal 
development objectives. The second is the use of technology to provide information or resources needed 
to support the workforce development process (e.g., provision of labor market information, career 
information). In this report, the term “TBL” will be used to describe either type of learning, and, where 
appropriate, the report will draw a distinction between the two forms of TBL, technology-based skills 
building and technology-based services. 


The Context for Technology-Based Learning 


The report begins by exploring the context for TBL at the state and local levels. In their respective 
surveys, SWA and Local Workforce Investment Area (LWIA) representatives were asked to describe the 
environmental factors against which TBL is adopted and used for workforce development. 


State Context for TBL 


SWAs, through their role in establishing policy and investment priorities, have the capacity to influence 
workforce development strategies and service delivery at the local level. The survey of SWA 
administrators collected information about their perceptions of TBL and its role in supporting the broader 
mission of the workforce system. The administrators’ responses suggest that states were establishing 
commitments and an infrastructure to support the use of TBL within the workforce system. Specifically: 
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Over half (60 percent) of SWA respondents rated TBL as a medium or high strategic priority. 


Approximately 40 percent of SWA respondents report that their states made institutional 
commitments to TBL by establishing policies, passing state legislation, or funding initiatives that 
specifically promote its use (Exhibit E.1). 


Nearly three-quarters (73 percent) of respondents reported that their SWAs use or encourage the use 
of TBL. Among those respondents, approximately 80 percent view TBL as a way of advancing other 
critical priorities such as increasing access to WIA services, expanding participant flexibility, and 
improving the overall cost efficiency of the system. 


The majority of SWA respondents (60 percent) report their states monitor the extent to which 
technology-based skills building and other modes of skills building were used by their Eligible 
Training Providers. 


Exhibit E.1: State institutional commitments to TBL 


ze) (AVA 
funding, or 
No policies, legislation, 
funding, or 44% 
legislation, 
56% 


Notes: n=45 
Additional detail provided in Exhibit A.3.3 in Appendix A 
Source: TBL state survey 


Local Implementation of TBL 


Against this backdrop, LWIBs made their own institutional investments that shape TBL opportunities and 
use at the local level. The use of TBL for workforce development at the local level has resulted in a 
delivery system characterized by services that rely on: 


Electronic technology only: participants can access services only through electronic technology (e.g., 
online, in a computer lab, videoconferencing); 


Blended delivery: participants can access services both through electronic technology and in person. 


In-person: participants can access services and activities by directly visiting AJCs or attending 
classes, workshops, or counseling sessions run in-person by service providers. 
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One-third (33 percent) of LWIA respondents reported that their AJCs offer some electronic-only services 
and another 79 percent reported that their AJCs offer various types of blended services (Exhibit E.2). 
Additionally, nearly one-fifth of the LWIA respondents report that their LWIBs invested in online-only 
providers, such as Virtual One-Stop, to provide technology-based services that supplement their 
traditional AJCs. 


Exhibit E.2: Delivery modes employed by American Job Centers 


100% 


80% 79% 


62% 
60% 


40% 
33% 


Percent of LWIA respondents 


20% 


0% 
Electronic Blended In-person 
AJCs’ delivery mode for core and intensive services 


All LWIA Respondents 


Notes: n=431 
Only respondents who reported that their LWIA had one or more Comprehensive American Job Center completed 


this question. 

Percentages do not sum to 100 since respondents could select more than one type of delivery mode. 
Additional detail provided in Exhibit A.4.3 in Appendix A 

Source: TBL local survey 


LWIBs made targeted investments designed to attract and support individual participants. The majority 
of respondents (72 percent) reported that their AJCs made proactive efforts to promote the use of TBL. 
While many report relying on traditional marketing approaches, over half of LWIA respondents reported 
that their AJCs were using some form of online promotional materials (63 percent) or social media (54 
percent) to advance their TBL agendas. To complement these marketing efforts, LWIBs made 
infrastructure investments that facilitate both access to and use of TBL. For example, nearly all LWIA 
respondents reported that their AJCs provide on-site access to computers labs, laptops, and the Internet, 
while more than half reported that their AJCs offer access to video or audio devices (63 and 54 percent, 
respectively). These investments may be used for both technology-based skills building and technology- 
based services. 


Finally, the majority of LWIA respondents reported that their AJCs were making investments in activities 
designed to maximize users’ comfort with, and efficient use of, technology-based platforms and content 
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(Exhibit E.3). For example, nearly two-thirds of respondents (65 percent) reported that their local 
delivery systems conduct an assessment or initial interview to determine participants’ levels of 
technology readiness. To supplement pre-program activities, two-thirds (66 percent) of LWIA 
respondents reported that their AJCs had some form of a technical assistance strategy to assist clients in 
using TBL. 


Exhibit E.3: Activities supporting the use of technology 


Percentage of LWIA 
respondents 

Assessment activities 
Assessment or interview to assess technology readiness 65 
Assessment or interview related to course hardware, software, or equipment 48 
requirements 
Support activities 
Technical assistance to support participants’ use of TBL 66 
Formal orientation to course technology or to learning management system 37 


Notes: N=349-372 
Additional detail provided in Exhibit A.4.8 in Appendix A 
Source: TBL local survey 


Integration of TBL into WIA Services 


AJCs serve adults, youth, and dislocated workers through the provision of WIA core and intensive 
services. They also administer funds for occupational training and provide referrals to adult education 
providers. In discussing these activities, the report distinguishes between the use of technology to 
promote and support core and intensive services and technology-based skills building that supports the 
development of academic, occupational, and/or personal skills. 


When asked about the use of technology-based services for providing core services, LWIA respondents 
reported that their AJCs generally use a blended approach, involving both in-person and electronic 
technology services. Approximately three-quarters (78 percent) of respondents indicated that their AJCs 
use a blended approach in their career exploration and job search process (for example, having staff 
introduce participants to online tools). However, many AJCs continue to provide exclusively 
personalized (in-person) core services in key information-based functions, such as the use of labor market 
information (45 percent) and exploration of career, training, and support service options (78 percent). 


Compared to core services, intensive services were somewhat less likely to have a technology component 
in their delivery. LWIA respondents identified the use of technology to support three categories of 
intensive services: 


e Foundational skills: Almost half (40 to 50 percent) of LWIA respondents indicated that their AJCs 
used a blended delivery approach to develop participants’ foundational skills (e.g., job application 
skills, computer software skills, soft skills). The lowest use of technology-based skills building was 
in the delivery of soft skills (37 percent blended delivery), which typically necessitate interpersonal 
support. 
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e Assessment: Compared to other intensive services, the assessment functions were more likely to be 
offered using blended delivery. Technology-based services were more prevalent in support of 
diagnostic testing (53 percent blended; 20 percent electronic-only) than in assessments designed to 
identify individual employment goals and barriers (35 percent blended; 12 percent electronic-only). 


e Counseling and case management: These intensive services were unlikely to be delivered using 
electronic technology (exclusively or blended) and were widely characterized by in-person delivery. 
This finding suggests that LWIA respondents acknowledge that in-person interaction remained a 
hallmark of these participant support activities. 


The personal computer (laptop or desktop) was the most prevalent form of technology used to deliver 
TBL-based core or intensive services. However, the data further indicated that smartphones (50 percent 
of LWIA respondents), as well as various video devices (53 percent of LWIA respondents) and 
videoconferencing (43 percent of LWIA respondents), were platforms through which participants may 
have accessed workforce development services. 


Unlike most core and intensive services, which are offered through AJCs, occupational training is offered 
primarily through eligible programs providers. To ensure a consistent reference point across the survey, 
LWIA respondents were asked to provide information on the use of TBL to support their five largest 
(highest number of participants with ITAs) training programs that served WIA participants during 
Program Year 2011 (PY 2011; July 1, 2011—June 30, 2012). Technology-based skills building in 
occupational training was being conducted using blended delivery models in one-quarter of the high- 
volume programs; few high-volume programs relied on exclusively distance models (Exhibit E.4). 


The use of technology-based skills building varied by characteristics of the training programs. 
Approximately half (54 percent) of the high-volume training programs were offered through public, post- 
secondary educational institutions, such as community colleges. Their rate of providing technology-based 
skills building was higher than the average, with nearly one-third (30 percent) of the LWIA respondents 
reporting that the colleges used either distance or blended options to deliver occupational training. The 
use of technology-based skills building as a training option varied by sector as well as credential offered. 
The highest use of distance learning or blended options was reported among training programs targeting 
IT occupations (47 percent) and those resulting in an Associate’s Degree (47 percent). 


Similarly, Title II adult education services are offered primarily through a network of providers. LWIA 
respondents were asked to provide information about the use of technology-based skills building to 
support services from up to five Title II providers to which their AJCs made the most referrals during PY 
2011. Approximately one-third of Title II service providers associated with the LWIAs responding to the 
survey delivered adult education services through technology-based skills building. Largely, providers 
using technology-based skills building were local education agencies, 35 percent of whom offer blended 
services. There was little variation in the extent to which service providers used technology-based skills 
building to deliver Title II services (adult basic education, general educational development (GED) 
classes, or English as a Second Language instruction). 
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Exhibit E.4: Delivery modes for high-volume training programs 


Distance, 1% ———__ 


Blended, 24% 


In-person, 
75% 


Notes: n=2,014 
Additional detail provided in Exhibit A.6.1 in Appendix A 
Source: TBL local survey 


Factors Affecting TBL Adoption and Use 


The integration of TBL into the workforce system is an evolving process that is affected by various 
factors. In their respective surveys, SWA administrators and LWIA representatives were asked to rate the 
importance of a number of factors in terms of whether they pose potential barriers to the adoption and use 
of TBL. 


Responses from the state survey suggest solid acceptance of the use of technology-based services, 
particularly among state policymakers and training providers. Over half of SWA respondents (56%) 
reported that state policymakers and training providers posed little or no barrier to implementing TBL 
(either technology-based skill building or technology-based services). SWA respondents did, however, 
express some concerns regarding both costs and the instructional effectiveness of TBL. 


SWA respondents reported concerns about actual expenditures on TBL and the organizational cost of 
transitioning to the use of TBL. Approximately 44 percent of SWA respondents reported that the cost or 
difficulty of implementing the necessary technology for TBL courses was a large or significant barrier. 
The cost or difficulty of developing TBL courses was also seen as a large or significant barrier by 38 
percent of SWA respondents. 


LWIA respondents’ perceptions of barriers were similar to those reported by SWA respondents. LWIA 
respondents also reported that key stakeholders’ (e.g., state policymakers, the local workforce system, 
employers) acceptance of TBL adoption and uses did not pose a significant barrier. Despite this receptive 
environment, three factors were reported as barriers that warrant further exploration and attention. The 
first barrier concerned costs. Approximately half of LWIA respondents cited the costs or difficulty of 
implementing TBL (e.g., assessing the quality of TBL courses, developing courses, implementing the 
necessary technology) as considerable barriers. Second, nearly two-thirds (64 percent) of LWIA 
respondents cited the participants’ levels of technological readiness as a significant barrier. A final factor 
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was the perceived limitations of TBL instruction, which almost half of LWIA respondents (42 percent) 
viewed as a large or significant barrier. This sentiment was underscored by the related perception that 
technology-based instruction may be less capable of fully engaging learners, resulting in lower levels of 
effort (e.g., social loafing; 51 percent of LWIA respondents) or course dropout (52 percent of LWIA 
respondents). 


Implications 


This research helped to identify several issues that should be further explored as ETA refines the focus of 
its TBL Initiative going forward. Overall, the research points to several inter-related factors related to 
current and future adoption and use of TBL in the workforce system. Areas for additional work include: 


e Understanding exactly what constitutes an appropriate level of technological literacy and where the 
current competency gaps are; 


e Promoting instructional effectiveness and consistency of TBL by working with providers and partners 
to further convey expectations or standards around TBL (e.g., course design and pilot testing, online 
teaching credentials, and availability of student support options); 


e Exploring institutional options to share developmental costs and defray operating costs of TBL; 


e Understanding the dynamics of TBL promotion in terms of what it takes to establish a technology- 
friendly service delivery environment, how to reduce dependence on or expectations of staff-based 
services, and how to use existing technology to promote newer technology; 


e Examining the role and effectiveness of technical assistance to: 1) provide the support needed for 
AJC participants to reap maximum benefit from WIA services and 2) create and communicate a 
supportive learning environment that will attract and retain participants in the future. 


Finally, the workforce system has a long established history of providing individualized and hands-on 
support to participants looking to enhance their employability and career opportunities. This research has 
demonstrated the prevalence of blended delivery options for both technology-based services and 
technology-based skills building. That emphasis on blended options underscores the importance of 
continuing to implement TBL at a pace and in a way that maintains the essential character of the 
workforce system. Moving forward requires that ETA’s TBL Initiative remains aware of, and sensitive 
to, establishing realistic and balanced expectations around the pace of TBL adoption. In addition it 
requires that ETA communicates an appropriately balanced message about TBL for policymakers, 
planners, practitioners, and providers. To do so, the ETA should continue to offer both the insight and 
support needed for state decision-makers and LWIBs to make informed decisions about where and when 
to invest in TBL within the workforce system. 
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Technology-based learning (TBL), also commonly known as e-learning, has gained considerable traction 
as a workforce development and training strategy. Broadly, TBL constitutes any learning via electronic 
technology. This definition includes online education as well as learning facilitated by other electronic 
technologies like Intranet sites, satellite broadcasts, audio- and videoconferencing, Internet bulletin boards 
and chat rooms, Webcasts, virtual classrooms, simulations, electronic gaming, podcasts, CD-ROMs, and a 
variety of mobile options (e.g., smart phone applications or “apps”). 


For purposes of this research, TBL is defined as an umbrella term that encompasses two types of learning 
via electronic technology. The first is the use of technology to promote skills building that supports 
academic, occupational training, or personal development objectives. The second is the use of technology 
to build knowledge or provide information or resources needed to support the workforce development 
process (e.g., provision of labor market information, career information). In this report, the term “TBL” 
will be used to describe either type of learning, and, where appropriate, the report will draw a distinction 
between the two forms of TBL, technology-based skills building and technology-based services. 


In addition to broadening accessibility, TBL may also allow for customization of content, delivery, and 
instructional approach. TBL-based applications may be predominantly instructor-centric (with an expert 
at the core who delivers a lecture), content-centric (learners interact individually with content embedded 
in a learning system), or participant-centric (the participant is the navigator, driving the content and pace). 
Finally, TBL delivery may also scale up more easily and cost-effectively than traditional classroom 
learning, allowing more learners to access content without a commensurate increase in resources. 


Given its potential for flexibility, ease of delivery, and cost-effectiveness, TBL may be especially 
applicable for use in the nation’s public workforce investment system. The Employment and Training 
Administration (ETA) of the U.S. Department of Labor 


(the Department) oversees the operation of key aspects of Technology-based learning (TBL), 
the workforce system and has supported the integration of also known as e-learning, constitutes 
TBL strategies into its service delivery system. any learning via electronic 
technology. This definition includes 
One key way ETA has supported the integration of TBL online education as well as learning 
for workforce development is through the TBL Initiative. facilitated by other electronic 
This study is part of that initiative, which aims to “increase technologies like Intranet sites, 
the number of people trained or certified in high-growth, satellite broadcasts, audio- and 
high-demand occupations and industries through the videoconferencing, Internet bulletin 
broadening of training opportunities for skills building boards and chat rooms, Webcasts, 
made available timely and conveniently through the use of simulations, electronic gaming, 
TBL methodologies” (U.S. DOL 2008b, 3). Sponsored by podcasts, CD-ROMs, and a variety 
ETA and conducted by Abt Associates, the study was of mobile options (e.g., smart phone 
designed to document and describe the extent to which applications or “apps”). 


TBL is currently being used, under what circumstances, 
and the factors that shape its adoption in the workforce system as a whole. It does so through the 
examination of two primary data sources: 


e A survey of state workforce agency (SWA) administrators in each state to gather insight into the state 
context for TBL adoption and promotion; and 
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e A survey of administrators of each Local Workforce Investment Area (LWIA) regarding the current 
use of TBL at the program level and factors that facilitate or impede its viability. 


Where possible, the research also sought to provide insight into specific program models that might be of 
value to those in the workforce system seeking to expand or refine their use of TBL. In-depth telephone 
interviews were conducted with LWIA respondents to gather additional detail on specific TBL efforts 
they referenced in completing their surveys. Ultimately, the findings from this research may be used by 
ETA to better understand the potential of technology-based skills building and technology-based services 
so that they can be most effectively integrated into the workforce system. 


The remainder of this chapter provides context for this study in two ways. First, it provides a very broad 
overview of the public workforce investment system, with an emphasis on the provision of services that 
may be particularly germane to TBL. This is followed by a brief overview of the ETA’s TBL Initiative, 
with an emphasis on select efforts that ETA has undertaken to promote the integration of TBL throughout 
the workforce system. Finally, it closes with an overview of the organization of the report. 


1.1 The Workforce Investment System 


To give context to this study, it is helpful to review the organizational structure of the workforce system. 
The workforce system was established by the Workforce Investment Act of 1998 (Public Law 105-220; 
WIA) and is a key resource for employment and training services for job seekers and businesses across 
the country. WIA is designed “to consolidate, coordinate, and improve employment, training, literacy, 
and vocational rehabilitation programs in the United States.”' Specifically, Title I: Workforce Investment 
Systems and Title I: Adult Education and Family Literacy mandate the provision of a range of workforce 
development services, including assessment, basic skills education, career readiness, occupational 
training, and job search. Title I focuses on the provision of workforce preparation and employment 
services designed to meet the needs of both businesses and (potential or incumbent) employees. For Title 
I, WIA authorizes activities at two levels within the workforce system: states and Local Workforce 
Investment Areas (LWIAs). Federal WIA funding is split such that 15 percent is reserved for statewide 
activities and the majority flows down to LWIAs.” 


WIA was designed to foster coordinated, long-term planning for workforce development programs.” It 
establishes American Job Centers (AJCs), formerly known as One-Stop Career Centers,“ as the 


The Workforce Investment Act of 1998 


This report focuses on services for adults and dislocated workers. Eight-five percent of adult funding is 
allocated to local areas and 15 percent is reserved for statewide activities. Dislocated worker funding is split so 
that 60 percent is allocated to local areas and 40 percent is reserved for the state (15 percent for statewide 
activities and up to 25 percent for state rapid response activities). 


For more information on the workforce investment system as established by WIA, see Barnow and King (2005). 
The long-term planning goal was intended, not only for WIA, but also for the Employment Service (labor 
exchange services supported under the Wagner-Peyser Act), and related funding streams such as Temporary 
Assistance for Needy Families (TANF) work programs, Adult Education and Family Literacy, Career and 
Technical Education, and Vocational Rehabilitation program. 


Training and Employment Guidance Letter 36-11 describes the implementation of identifying or branding all 
One-Stop Career Centers as American Job Centers (U.S. DOL, 2012a).° Any adult age 18 or older does not 
need to meet any qualifying characteristics in order to receive core services. 
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cornerstone and front end of the local workforce delivery system. At the AJCs, services for job seekers 
are sequenced through three tiers: core, intensive, and training services. 


e Core services are universally available° and generally involve the provision of information on job 
openings; the labor market; and providers of training services, youth activities, adult education, 
vocational rehabilitation activities, and vocational education. 


e Intensive services are available for individuals who are unable, after receiving core services, “to 
obtain or retain employment that allows for self-sufficiency.”° These services generally involve 
individualized activities such as counseling and assessment to help customers choose training 
programs and select occupational areas. 


e Training generally consists of classroom or on-the-job preparation for a specific occupation or set of 
occupations and, if funding is available, is open to individuals who are unable to obtain or retain 
employment after receiving intensive services. Voucher-based Individual Training Accounts (ITAs) 
are used to allow individuals to choose among a variety of approved training providers. Local 
Workforce Investment Boards (LWIBs)—under guidance from the states—are responsible for 
establishing and overseeing the Eligible Training Provider List (ETPL), which lists, for each state, the 
approved training programs from which participants can access WIA services using ITAs. 


Additionally, individuals who do not have a high school diploma may be referred by the AJCs to adult 
education services. Adult education services are described by WIA Title II and are designed to ensure 
that participants have the educational foundation needed to pursue employment opportunities that can 
lead to economic self-sufficiency. Title II services are administered by the U.S. Department of 
Education’s Office of Vocational and Adult Education (OVAE). 


At the state level, WIA mandated the creation of state Workforce Investment Boards (WIBs). Each Board 
comprises the governor, two members of each chamber of the state legislature, and representatives 
appointed by the governor, including business leaders, chief elected officials, labor organization leaders, 
state agency heads, and individuals with related experience. The Board’s duties include providing broad 
policy guidance to the governor on the statewide workforce system and statewide labor market 
information system. The Board also helps the governors monitor statewide activities and provide grant 
planning and operations infrastructure for regional or local WIBs. 


While states vary in their organizational structures and divisions of responsibilities, generally an SWA 
(e.g., state departments or offices of labor, commerce, or workforce development) may work with the 
state WIB on statewide employment and training activities. SWAs tend to be charged with administering 
and disbursing WIA Title I funds. Their specific responsibilities relevant to TBL may include assisting 
the state WIB in overseeing LWIBs, overseeing the operation of the AJCs, and disseminating the state list 
of approved training programs. 


Within each state, LWIBs are funded under WIA Title I and appointed by local elected officials in 
accordance with criteria established by the relevant governor. They must include representatives from 


° Any adult age 18 or older does not need to meet any qualifying characteristics in order to receive core services. 


° If funding is limited, LWIBs, in consultation with their state’s governor, prioritize providing intensive and 


training services to recipients of public assistance and other low-income individuals. 
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business, education providers, labor organizations, community-based organizations, and economic 
development agencies. LWIBs oversee each local workforce investment area (LWIA). 


LWIAs are geographical jurisdictions within states that are designated by each state governor. LWIAs 
tend to be geographic areas served by common educational institutions and consistent with local labor 
market and labor pool areas. Local governmental units with populations of 500,000 or more tend to 
constitute their own LWIAs. LWIBs designate intermediaries to provide WIA services within their 
LWIAs; LWIBs themselves are rarely service providers. 


The intermediaries are public or private organizations that operate the AJCs, training programs, and 
providers of Title II services. AJCs’ responsibilities include providing core and intensive services and 
administering funds for direct training services. They are also required to refer qualifying participants to 
Title II providers for adult education services. It is this bundle of WIA Title I and Title II services, and 
the prevalence of TBL in providing these services, that is the focus of this report. 


1.2. ETA’s Technology-Based Learning Initiative 


Since introducing the TBL Initiative, ETA has played a leading role in promoting the use of TBL in the 
workforce system (U.S. DOL 2008b, 3). What began as internal discussions within ETA around the use 
of TBL has developed into a national initiative within the workforce system with the release of the 
Training and Employment Guidance Letter (TEGL) No. 17.07, Using Technology-Based Learning in the 
Workforce Investment System, in 2008. ETA’s multi-pronged strategy has consisted of a number of 
complementary projects to inform the field about TBL, share best practices for incorporating technology 
into the workforce system, and invest in and evaluate promising programs using technology-based 
services and technology-based skills building strategies. The initiative is evolving through ongoing work. 
For example, through their investments in the Trade Adjustment Assistance Community College and 
Career Training (TAACCCT) Grants Program, the Department placed an emphasis on “leveraging 
advanced training technologies that go beyond traditional online education and accelerate learning and 
credential attainment through a variety of means...[including] skillful use of technology-enabled new, 
emergent models of online education and workforce training, such as Massive Open Online Courses 
(MOOCs); prior learning assessments; and online and blended learning environments that allow larger 
numbers of workers to participate.”’ To date, many of the efforts undertaken as part of ETA’s TBL 
Initiative have focused primarily on strengthening the technological component of occupational training. 
The Initiative’s learning technologies first encompassed various forms of distance and online learning. 
Over time, however, technology has continued to evolve at a rapid pace, which has further extended the 
reach of TBL. Such developments as the large-scale adoption of mobile technology, expansion of 
broadband and wireless networks, emergence of digital natives who do not know a world without 
computers, adoption of natural human interfaces on tablets and smartphones, and the spread of Internet 
content creators (e.g., bloggers) have increased the integration of technology into everyday life. These 
changes in technology may affect its application within the realm of both technology-based skills building 
and technology-based services. 


7 U.S. DOL, 2013a 
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Within this highly dynamic context, ETA has expanded its horizons regarding the scope and potential of 
TBL.° Previous research primarily has examined technology-based skills building, that is, the use of 
technology to expand the accessibility and flexibility of interventions that promote traditional academic 
and occupational skills.” This focus is the natural byproduct of considering TBL as an alternative to 
traditional classroom-based instruction. This focus on technology-based skills building was the emphasis 
of the 2008 TEGL that addressed the use of TBL in the workforce system.’ In that TEGL, ETA noted 
that “WIA and ETA’s policies and guidelines do not prohibit the use of technology to provide training. In 
fact, WIA’s guiding principles for the development of workforce investment programs encourage local 
flexibility and innovation for providing training options.” 


However, there is nothing about TBL that limits its application to the structured context of occupational 
training or education. This study recognizes that learning occurs more widely for participants in the 
workforce system and electronic technology can, and does, contribute to that learning. Specifically, 
within the workforce system there are a wide variety of employment and training activities, structured and 
unstructured, where TBL in the form of technology-based services may be used to build knowledge 
among participants. For instance, 


e Learning about oneself through computerized assessments; 


e Learning about career and training opportunities through online labor market information systems or 
other databases of career and technical information; 


e Learning appropriate behaviors required on the job through video tutorials on soft skills and post- 
placement support via video chats; 


e Learning about connecting to virtual community support resources as part of the case management 
process; and 


e Learning about the best strategies for attaining successful employment through social networking 
online with job development and placement service providers. 


Further, the use of technology-based services may build technological literacy and skills that may 
contribute to future engagement with TBL. 


Finally, in addition to facilitating the achievement of various learning objectives, technology has the 
potential to contribute to a range of services provided by the workforce system, from outreach to 
information sharing to job placement. Many of these functions fall under the heading of core and 
intensive services, which are used by the largest proportion of WIA participants. One example that 
touches many aspects of the service menu is O*NET, a free, online database and tool for career 
exploration and job analysis (www.onetonline.org). The provision of career exploration and job 
information is among the set of core services mandated by WIA and typically is not delivered in a 


For example, in Training and Employment Notice (TEN) 29-12 from April 3, 2013, ETA announced new 
features and enhancements to online electronic tools to assist individuals in exploring careers and preparing for 
and finding employment. http://wdr.doleta.gov/directives/attach/TEN/TEN_29_12.pdf 


For example, Dunham, Estrella, and Nyborg (2011); Dunham, Estrella, Leufgen, and Henderson-Frakes (2011); 
and Maxwell, Sattar, Rotz, and Dunham (2013) 


© TEGL 17-17, p.4 
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structured classroom setting. It is available exclusively online using basic electronic technology to enable 
users to customize their searches for career information. It similarly integrates an assessment function to 
match users’ interests with careers for further exploration. 


This research reflects the broader perspective on TBL by examining technological content across an array 
of workforce investment services. As appropriate, the report draws a distinction between technology- 
based skills building, which directly supports academic, occupational training, or personal development 
objectives, and technology-based services, which primarily build knowledge or provide information or 
resources needed to support the workforce development process. 


1.3 Structure of the Report 


The remainder of the report is organized as follows: Chapter 2 presents the research questions guiding the 
study and the methodology that was used to collect data in the two surveys. Chapter 3 discusses 
institutional support for TBL at the state level. Chapter 4 describes the context for TBL use at the local 
level. Chapter 5 examines how the AJCs use technology-based services and technology-based skills 
building to provide AJC participants with Title I core and intensive services. Chapter 6 documents the 
use of technology-based skills building for Title I training services, and Chapter 7 examines technology- 
based skills building for Title II services. Chapter 8 describes SWA and LWIA respondents’ perceptions 
of key factors underlying the adoption and use of TBL. The report closes with a summary of study 
results, followed by implications in Chapter 9. 
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2. Study Approach and Data Sources 


In recent years, the ETA has supported the integration of TBL strategies in workforce system services. 
However, little is known about the extent to which these strategies are being used nationally and under 
what circumstances and the factors that shape their adoption in the broader field. This study surveyed 
workforce system policymakers and practitioners regarding the use of TBL for employment, training, and 
education activities and the factors that influence the policies, decisions, and practices concerning TBL. 
Specifically, this research was designed to: 


e Identify the extent to which states placed a policy or program emphasis on the use of technology- 
based skills building and technology-based services; 


e Explore local program factors that may facilitate or impede decisions to integrate TBL into 
employment, training, and education activities; and 


e Expand knowledge of the current use of TBL in the workforce system, including, 


— The current types of platforms, instructional approaches, and technologies currently in use for 
Title I core and intensive services, 


— The extent to which technology-based skills building is currently being used by approved training 
providers, and 


— The extent to which technology-based skills building is currently being used by adult education 
providers. 


This chapter provides an overview of the study approach and data sources. It begins by discussing the 
research questions that shaped and organized the work. It then documents the approach to addressing 
those questions: (1) the study respondents; (2) instrument development; (3) survey administration and 
response rates; (4) analysis methods employed; and (5) limitations of the data and analysis. Further 
details on the methodology, including the data collection instruments, are included in Appendix B. 


2.1 Key Research Questions for the Study 


This study was designed to develop an understanding of the use of TBL in the workforce system. To 
address this issue, the study focused on two related research questions: 


What is the current extent and nature of TBL usage in the public workforce investment system? 


This first question examines the current scope and mix of TBL usage within the workforce system, with a 
focus on TBL use for employment, training, and education activities. To understand current usage, Abt 
examined TBL use over the period of one year, from a variety of perspectives. This documented the 
extent to which technology-based skills building and technology-based services were provided both 
within and across LWIAs, including TBL delivery (the mix of providers, platforms, approaches and 
technologies) and the content areas it addressed. In particular, this study focused on several types of core, 
intensive, and training services provided under Title I, and the adult education services provided under 
Title II. 
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What factors appear to influence TBL usage in the public workforce investment system? 


The study also examined variation in TBL usage, particularly the policies and program factors that may 
affect the types and frequency of use. There are two parts to this question: understanding the context in 
which TBL is used and understanding practitioners’ perceptions of and experience with TBL. First, the 
study examined state-level policies and supports that may influence TBL adoption and usage at the 
program level. This question focused on resources and the policies, regulations, and legislative contexts 
surrounding TBL use. Collectively, these factors shape the extent to which local delivery systems invest 
in TBL as part of their service delivery strategies. The study also examined these issues at the local level. 
As discussed, LWIBs, through AJCs and other providers, implement WIA Title I and II services. 
Integration of TBL into local services may vary considerably depending on infrastructure and resource 
availability, types of programs, and the technological readiness and receptivity of the service population. 


2.2 Data Collection Approaches 


Two methods were used to collect data to address the study’s research questions. These methods are: 


e A survey of state-level workforce system representatives that examined the policies and supports that 
influence TBL adoption and use, and 


e A survey of local-level workforce system representatives that examined local TBL implementation. 


To maximize the scope and depth of the study, surveys were administered to the universe of states and 
LWIAs. The local-level survey was followed by in-depth phone interviews with select respondents to 
gain additional detail on their answers." 


2.2.1 Respondent population 


To ensure consistency in the administration of the state and local surveys, primary respondents were 
identified at each level who could consistently comment (or solicit information from relevant colleagues 
or stakeholders) in response to the range of questions in the surveys. 


State workforce system representatives. SWA administrators were the primary respondents for the 
state-level survey. This population was most likely to be knowledgeable about the context for TBL and to 
have a broad perspective on its implementation within the states’ workforce systems. In particular, while 
states vary in their organizational structures and divisions of responsibilities, generally an SWA (e.g., a 
state department of labor) and/or the state WIB is responsible for statewide employment and training 
activities. Additionally, SWAs may also influence the context for TBL at the state level by implementing 
policy, providing resources or technical assistance to LWIAs, or working with businesses. Where 
appropriate, state workforce agency administrators consulted with their state WIBs, their staff, or data 
systems to complete their surveys. 


"See chapter 4 for additional information on these respondents. 


' For the purposes of this research “state” refers to any area in the United States with a state WIB, i.e., the 50 


federated states and the territories of Washington DC, Guam, Puerto Rico, the Virgin Islands, Micronesia, the 
Marshall Islands, and the Mariana Islands. 
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Local workforce system representatives.’ LWIB Executive Directors were the primary respondents 
for the local-level survey. The LWIBs are broadly responsible for planning and designing workforce 
development strategies, including TBL. This is done within the context of broader state policies, as well 
as priorities established within the LWIA. These stakeholders are positioned to understand the breadth 
and mix of training and developmental resources within their LWIA and the extent to which that may 
have been (or can be) marshaled to support TBL. Where appropriate, LWIB Executive Directors drew 
upon knowledge of local delivery system staff, training and Title II service providers, or their data 
systems to answer their surveys. 


2.2.2 Instrument development 


Separate surveys were developed for each of the two respondent populations. Despite the separate data 
collection tracks, the two instruments were designed together to ensure comparability and integration of 
analysis and interpretation where feasible. The organization and content of both instruments are 
summarized in the Exhibit 2.1. Copies of both surveys are available in Appendix C. 


Both instruments were primarily designed for online administration. However, the surveys were also 
available for administration by phone and hard copy for those who preferred not to complete the online 
version. The instruments were field tested and refined prior to their administration. Pretesting was 
conducted with members of the primary respondent population who were selected with input from 
members of the appropriate Regional Offices.'* Pretest respondents completed the survey instruments, 
then provided structured feedback on the survey to the study team. The state survey was pre-tested with 5 
respondents; the local survey was pretested with 9 respondents. 


‘3 Of the 580 LWIAs, most are overseen by LWIBs and LWIB Executive Directors are the primary respondent for 


those surveys. In 16 states, LWIAs are overseen by state WIBs because there are no local WIBs. In these 16 
states, state WIB Executive Directors are considered as equivalent to the local WIB administrators, are the 
primary respondent to the “local” survey, and are referred to as LWIB Executive Directors in this report. In 16 
states, LWIAs are overseen by state WIBs: According to the CareerOneStop website (www.servicelocator.org), 
in AK, DC, DE, FM, GU, ID, MH, MP, MT, ND, NH, SD, UT, VI, and WY, the state WIB oversees all state 
LWIAs because there are no local WIBs. In AL, the SWA oversees the LWIA for some state counties. 


‘4 DOL’s team of Regional Administrators was asked to identify SWA administrators, state WIB chairs, and 


LWIB Executive Directors and Chairs as potential pre-test respondents within their regions. The state survey 
was pre-tested with respondents from each of the five state respondents recommended by the Regional 
Administrators. These five state respondents represented five (of the six) workforce regions. Twenty-two 
potential respondents were recommended to pre-test the local survey. Subsequent to a review of key 
characteristics and services, nine were selected to maximize geographic and program diversity. The first three 
interviews were conducted to inform broad measurement areas and constructs, and familiarity with key terms 
and content areas. Once the survey was fully drafted, Abt conducted a more traditional pre-test with the 
remaining six LWIA representatives. Collectively, these six LWIAs represented each of the six workforce 
regions. 
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Exhibit 2.1: Summary of the survey instruments 


TBL state survey 


Primary respondents: SWA administrators 
Estimated length: 30 minutes 
Content: 


e State policy, guidance, and support for TBL: Questions about the general context for TBL within 
the state, including measures of institutional commitment”? to TBL 


e TBL and the state Eligible Training Provider List (ETPL): Questions about the use of online or 
distance learning to support eligible training programs within the state 


e TBL challenges or barriers: Questions about the major factors affecting TBL implementation in 
the state 


TBL local survey 


Primary respondents: LWIB Executive Directors 

Estimated length: 60-90 minutes 

Content: 

e The LWIA: Questions about the composition of the local AJCs 


e WIA Title | core and intensive services: Questions about the use of TBL, particularly technology- 
based services, to support WIA Title | core and intensive services provided by the local AJCs 


e WIA Title | training services: Questions about the use of TBL, particularly technology-based skills 
building, to support training programs receiving WIA funds 


e WIA Title Il services: Questions about the use of TBL, particularly technology-based skills 
building, to support Title Il service providers to whom the local delivery system makes referrals 


e TBL challenges or barriers: Questions about the major factors affecting TBL implementation in 
the LWIA? 


2.2.3. Survey administration and response 


The surveys were fielded during an eight-week period in spring 2013. Contact information for all 
respondents, including e-mail addresses and phone numbers, were provided by DOL from the America’s 
Service Locator database.'® This database, which is accessible to the public, contains contact information 
for organizations associated with the workforce system and is updated regularly by staff at ETA. 


To notify respondents of the survey, an advisory was sent from the Assistant Secretary of ETA to all 
primary respondents, as well as ETA Regional Administrators, state and local workforce agencies and 
WIBs, and all state WIB Executive Directors (Training and Employment Notice 21-12; U.S. DOL, 
2013b). Primary respondents were then sent an e-mail invitation to participate in the survey that 
contained a live personalized hyperlink to the online instrument (see Appendix C for communications 
about the survey and the survey instruments). An in-house survey solutions desk was staffed to handle 


Henceforth, we use the terms “institutional commitment” or “buy-in” to refer to states that reported they had 


policies, funding, or legislation specific to TBL within their state workforce investment systems. See Chapter 3 
for additional information. 


16 www.ServiceLocator.org 
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inquiries from the field, provide reminders and encourage survey completion, administer the instrument 
via phone, and conduct follow-up, as necessary. 


Exhibit 2.2 summarizes the total number of each survey that was distributed and the rates of response. 
The state survey was fielded to 57 representatives of SWAs, and the local survey was fielded to 580 
LWIA representatives. A total of 47 state surveys and 399 local surveys were completed by respondents, 
respectively.'’ The response rate is 81 percent for the state survey and 69 percent for the local survey. ’® 
The cooperation rate—the proportion of the survey sample that provided complete or partially complete 
responses—is 84 percent for the state survey and 76 percent for the local survey. The analysis included 
all respondents who provided data for the relevant survey item(s), regardless of whether they provided 
complete or partially complete responses. 


Exhibit 2.2: State and local survey response 


State survey Local survey 

Number of units 

Survey sample 57 580 

Partially complete and complete 49 443 

Complete 47 399 
Percent of units 

Cooperation rate 86% 76% 

Response rate 82% 69% 


2.2.4 Analytic strategy 


For variables with categorical responses,’ the report presents unweighted percentages. For items using a 
continuous response,” the report presents summary statistics on both the average and distribution 
(including histograms to summarize distributions and using the top and bottom quintiles to create useful 
categories). 


In addition to describing the prevalence of TBL in the state workforce system overall, Chapter 3 provides 
a comparison of the prevalence of TBL in states that did and did not report institutional commitments to 
TBL. To indicate if these differences are meaningful or may be due to chance, the analytic exhibits 
include an asterisk if the difference is statistically significant at the 10 percent level based on the results of 
t-tests when comparing two proportions and on Fischer’s exact test when comparing sets of categories.” 


'” For a survey to be considered complete, the respondent must have provided data for all items presented by the 


web survey. 


'® Appendix A includes additional information on the analytic sample sizes for each exhibit in the report. 


'9 For example: a) Electronic-technology only, b) blended, or c) in-person only. 


°0 In other words, a numerical response such as the percentage of LWIA participants who took part in any TBL in 


program year 2011. 


*1 Although the survey sample includes the universe of SWAs, the final analytic sample includes only survey 


respondents (including those with partially completed surveys). Hypothesis testing investigates whether the 
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Similarly, although the focus of Chapters 4 and 5 is on describing the prevalence of TBL at the local 
level, these chapters also describe the characteristics of LWIAs that were in the top quintile and bottom 
quintile of LWIAs in terms of the percentage of LWIA participants who participate in any TBL. Since 
the subgroups do not span the entire analytic sample, each subgroup is compared to the remainder of the 
sample separately.” An asterisk indicates the statistical significance of each of these comparisons 
separately at the 10 percent level based on the results of t-tests when comparing two proportions and or 
chi-squared tests when comparing sets of categories. 


Throughout this report, the number of observations included in the analysis varies based on the number of 
respondents who provided substantive responses to a particular survey item. To include the maximum 
sample size possible, the analysis includes all respondents who provided data for the relevant data item(s) 
regardless of the completion status of their surveys. Additionally, many survey items were not fielded to 
all respondents based on responses to prior survey questions (e.g., respondents were not asked about the 
mode of delivery for a particular service if that service was not provided). Finally, the analysis excludes 
respondents who responded “Unknown” to a particular data element. Appendix A provides details on the 
categories and numbers of respondents excluded from each analysis. 


2.3. Study Limitations 
Several limitations of this study should be noted: 


The analysis presented in this report is descriptive and the results should not be interpreted as providing 
information about causal relationships. It is important to note that, given ongoing changes in the 
workforce system, these data represent a snapshot of the use of technology-based skills building and 
technology-based services in the workforce system at a single point in time. However, the results may be 
used to develop hypotheses that can be examined using a rigorous experimental or quasi-experimental 
design. 


Although the state and local surveys were fielded to representatives from the universe of states and 
LWIAs and the response rates were 81 and 69 percent, respectively, it is possible that the survey data are 
affected by nonresponse bias. That is, the characteristics of the states and LWIAs in our analytic sample 
may differ systematically from the characteristics of the states and LWIAs in the population at large.” 


While respondents were encouraged to consult with other stakeholders and data sources as needed, these 
data are limited by respondents’ knowledge, recall, and perceptions. Each survey included questions 
about the sources respondents consulted to complete the survey and many respondents cited additional 
sources, indicating that the responses may be reliable reports of activities within the workforce system. 
The surveys used a broad definition of TBL and respondents’ knowledge, recall, and perceptions about 


differences observed in the final analytic sample can be reasonably expected to generalize to the universe (see, 
for example, Casella and Berger 2002) 


*2 The analysis adopts this strategy of separate comparisons for substantive reasons (see, for example, Casella and 


Berger 2002). The variables that characterize LWIAs in the top quintile of TBL participation from the 
remaining LWIAs may be quite different from those that characterize LWIAs in the bottom quintile. This 
analytic approach allows us to separately consider what variables differentiate each of the subgroups from the 
general population of LWIAs. 


*3 See Appendix B for the non-response analysis. 
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what constitutes TBL. Where possible, the report draws distinctions between technology-based services 
and technology-based skills building. However, in many instances, the data have limited ability to 
separate the two. 


Finally, since the purpose of the study was to describe the workforce system overall rather than to 
characterize individual cases (e.g., for performance measurement or auditing purposes), no quantitative 
data are presented on individual respondents or LWIAs. Qualitative responses are linked to respondents 
only when they gave permission to do so. Further, the analysis does not describe the use and adoption of 
TBL for subgroups of LWIAs defined by geographic factors (e.g., rural service areas, broadband 
coverage). Such analyses may be promising avenues for future research. 
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This chapter draws on the unique perspective of the SWA administrators who responded to questions 
about both the policy and operational contexts that shape the use of TBL in their states’ workforce 
systems. The designated SWA is charged with administering and dispersing WIA funds through the 
network of LWIBs in their states. In this capacity, SWAs have the potential to influence the use of TBL 
through policies, priorities, and direct investments in, for instance, infrastructure or technical assistance. 


An examination of state policies, strategic priorities, and institutional commitments provides a backdrop 
for understanding the adoption of TBL within local workforce systems. For example, the composition of 
a state’s ETPL broadly creates parameters for the integration of technology-based skills building into the 
delivery of training opportunities at the AJCs. SWA respondents’ insights into state contexts for TBL 
serve as important markers of TBL integration into the workforce system as a whole. 


To begin, this chapter discusses the prioritization of TBL at the state level, and explores some of factors 
that contribute to the level of priority it receives. It then describes the types of state-level support for 
TBL, particularly the extent of institutional commitments to TBL as evidenced by policies, legislation, 
resource allocation, and current use. The chapter concludes with a discussion of the statewide availability 
of TBL training programs. 


3.1 Prioritization of TBL at the State Level 


As shown in Exhibit 3.1, SWA respondents in most states considered TBL use to be a priority (84 
percent). Nearly 1 in 10 (9 percent) reported this was a high priority. An additional half (51 percent) 
reported that the use of TBL was a medium priority to their SWA. Finally, 24 percent reported that TBL 
use was a low priority and less than one-sixth of respondents (16 percent) reported that it was not a 
priority in their state. 


Exhibit 3.1: Priority for the use of TBL in the state workforce system 


Not a priority, 
16% 


Low, 24% 
Medium, 51% 


Notes: n=45 
Additional detail provided in Exhibit A.3.1 in Appendix A 
Source: TBL state survey 
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While these reported priority levels provide preliminary insight into states’ focus on TBL, SWA 
respondents were also asked to expand on their answers. First, SWA respondents who considered TBL 
use a high priority within their state were asked to explain their rating. Overall, their reflections 
emphasize TBL’s importance in improving the reach of workforce system services. For example: 


e [Using TBL] decreases the time from unemployment to credential to employment. 


e Learning delivery is no longer taught and received in what has been the norm (face-to- face settings) 
but [is now taught and received] with the integration of [all types of] technology. 


e [Our SWA] has prioritized the use of TBL to meet the increased demand for re-employment services 
for job seeking customers. 


Conversely, SWA respondents who considered TBL a lesser priority were asked a structured question 
about the reasons behind their ratings (Exhibit 3.2). The two most commonly selected reasons reflected 
the relative importance of TBL and responsibility for TBL. Specifically, nearly 4 in 10 respondents (38 
percent) indicated that other issues were of greater importance.~* The same proportion (38 percent) noted 
that TBL was considered a local-level (rather than state-level) strategic issue.” 


Exhibit 3.2: Reasons for states’ priority level for TBL, if it was not a high priority 


Percentage of 


state 
Reason for priority level respondents 
Other initiatives were higher priorities 38 
TBL was considered a local-level strategic issue 38 
TBL was considered already widely used within the state’s workforce system 20 
TBL was not discussed or considered as a possible component of the state’s 18 
workforce system 
TBL was considered not appropriate for the state’s workforce system 3 
Other 10 


Notes: n=40 

Only states that responded that TBL was “Medium,” “Low,” or “Not a priority” were asked this question. 
Percentages do not sum to 100 since respondents could select more than one response. 

Additional detail provided in Exhibit A.3.2 in Appendix A 

Source: TBL state survey 


An additional one-fifth (20 percent) of SWA respondents reported that TBL was a lesser priority because 
it was already widely used within their workforce system. The remaining responses reflect a lesser 
degree of awareness and consideration. Challenges to TBL implementation remain for the approximately 
one-fifth (18 percent) of the respondents who reported that TBL was not discussed or considered, and an 
additional 3 percent felt that it was simply not an appropriate component of the state workforce system. 


4 E.g., veteran’s employment issues, expanding focus on career pathways, and long term unemployment. 


*’ To further explore the concept of TBL as a local-level strategic issue, the local context of TBL and local-level 


use of TBL will be discussed further in Chapters 4 and 5 from the perspective of LWIA representatives. 
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However, these data as a whole suggest that overall awareness of TBL exists and was a relatively 
important strategy for SWA respondents. 


3.2 Institutional Commitments to TBL 


While the data above suggest that TBL has achieved a broad base of awareness and consideration at the 
state level, this may not necessarily translate into adoption and ongoing willingness to invest in the 
approach. This section examines several measures that, collectively, provide evidence of states’ 
commitments to TBL. 


Committing to TBL through state legislation 


Washington’s State Bill 6295 (of 2008) is an Act aimed at promoting technology-based skills building by 
creating workplace-based electronically distributed learning opportunities. It includes: 


e A demonstration program to develop and evaluate employer workplace-based educational programs 
with distance learning components 


A study of such programs nationally to understand standards and best practices for increasing student 
access and increasing the supply of open course materials, including digital open textbooks 


SWA administrators were asked a series of questions to gauge the nature and extent of state-level 
commitments to TBL. In particular, respondents were asked whether or not their states had policies 
specific to TBL, allocated funding for TBL initiatives, or had enacted legislation specific to TBL within 
their states’ workforce systems (Exhibit 3.3). Collectively, these data provide evidence of each state’s 
institutional commitments to TBL.”° 


Exhibit 3.3: States’ institutional commitments to TBL 


Percentage of 


Institutional commitments to TBL ener 
States with institutional buy-in to TBL 

States with policies, funding, or legislation for TBL initiatives 44 
Types of institutional buy-in to TBL 

Policies specific to TBL within the state workforce system 32 
Funding for TBL workforce investment initiatives 19 
Legislation specific to TBL within the state workforce system 2 


Notes: n=43-48 

Percentages do not sum to 100 since respondents could select more than one response. 
Additional detail provided in Exhibit A.3.3 in Appendix A 

Source: TBL state survey 


°° Note that states that responded “Unknown” to all three items were categorized as not demonstrating institutional 


commitments to TBL. 
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In total, 44 percent of the responding states demonstrated their institutional buy-in to TBL through one or 
more of these actions.”” Specifically, nearly one in five (19 percent) SWA respondents reported that their 
SWA funded some type of TBL workforce initiative. Additionally, almost one-third (32 percent) reported 
that their states had policies specific to TBL within their workforce system. Finally, a minority (2 
percent) of states had legislation specific to TBL in place. 


Exhibit 3.4: Direct funding for TBL 


Percentage of 


state 
Direct funding for TBL respondents 
Funding for TBL initiatives 
State workforce agency funded TBL workforce initiatives 19 
Groups eligible to receive direct funding 
Training services providers 57 
Local workforce investment boards 43 
Local AJCs 29 
Businesses 29 
AJC participants 29 
Source of direct funding for TBL 
Federal 
Federal workforce funds (e.g., Workforce Investment Act or Wagner-Peyser funds) 71 
Recovery Act (ARRA) funds, TAACCCT grant, Pathways out of Poverty grant, H-1B 33 
Technical Skills Training grant 
Postsecondary education grants, loans, or scholarships 0 
Other federal funds 29 
State 
State tax deductions or credits 0 
Other state funds 43 
Other funding sources 
Grants from private organizations 17 
Employer-sponsored funding (e.g., tuition reimbursement) 14 
Loans from private lenders (e.g., banks) 0 
Reduced tuition or fees 0 
Other 40 


Notes: Funding for TBL initiatives, n=47. 

Only respondents who reported that their SWA funded TBL workforce investment initiatives responded to questions 
regarding the groups eligible to receive funding and the source of funding. Groups eligible to receive direct funding, 
n=7. Source of direct funding, n=5-7. 

Percentages do not sum to 100 because respondents could indicate multiple responses. 

Additional detail provided in Exhibit A.3.4 in Appendix A 

Source: TBL state survey 


°7 The three types of support for TBL were highly related. All three items (policy, funding, and legislation) were 


significantly correlated (p<0.01) with pairwise Pearson’s correlation coefficients ranging from 0.43 to 0.64 (see 
Exhibit A.3.3a in Appendix A). The statistically significant results of the correlation test provide reasonable 
evidence that the three measures can be combined into a valid single measure. 
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Using this definition, the remaining 56 percent of the responding states lacked this demonstrated level of 
commitment. The distinction between states that demonstrated institutional commitments and those that 
did not provides an opportunity to explore potential differences across these two broad groups. 
Throughout this chapter, findings are compared between these two categories of states. To indicate if 
these differences are meaningful or may be due to chance, the analytic exhibits include an asterisk if the 
difference is statistically significant at the 10 percent level based on the results of t-tests for differences in 
proportions and on Fischer’s exact test when comparing sets of categories.” 


Providing resources to fund TBL initiatives represents a notable commitment to TBL. Across the 19 
percent of states that provided this type of support, there was considerable variation in both the recipients 
and the sources of those funds. Exhibit 3.4 indicates that the majority of states providing funding (57 
percent) reported that training providers were eligible to receive these resources. In addition, the most 
commonly cited source of funding for state-level initiatives was Federal workforce resources such as 
WIA or Wagner-Peyser funds (71 percent). 


3.3 Additional Statewide Support for TBL 


In addition to the broad institutional commitments discussed above, states may opt to promote TBL use 
through other channels as well. These channels may include outreach, marketing and promotion, or the 
targeting of specific resources and investments. 


SWA administrators were asked about their agency’s use or promotion of TBL to complement other 
statewide objectives. As shown in Exhibit 3.5, nearly three-quarters of respondents (73 percent) reported 
that their states used or encouraged the use of TBL as a way of advancing other critical statewide 
objectives. Using PY 2011 as a reference point, these respondents cited TBL’s importance in increasing 
the accessibility of WIA services and increasing individual options and personal flexibility in pursuing 
WIA services (81 and 80 percent, respectively). Similarly, a majority of SWA respondents reported using 
or encouraging TBL use for improving cost efficiency in WIA delivery (79 percent), attracting priority 
target groups (70 percent), and promoting coordination of services across agencies (68 percent). 


Exhibit 3.5: SWA used or encouraged use of TBL 


Percentage of | Percentage of 
states states not 
Percentage of reporting reporting 
state institutional institutional 
respondents buy-in buy-in 
SWA used or encouraged use of TBL 73 80 68 


Notes: All states, n=45 

States reporting institutional buy-in, n=20 

States not reporting institutional buy-in, n=25 

* Indicates that the difference between states that do and do not report institutional buy-in is statistically significant at 
the 10 percent level. 

Additional detail provided in Exhibit A.3.5 in Appendix A 

Source: TBL state survey 


*8 See Chapter 2 for additional information on the study’s methodology. 
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As also shown on Exhibit 3.6, most of these findings were consistent across states, regardless of their 
levels of institutional commitment to TBL. There are no significant differences between states in either 
their use or encouraged use of TBL. However, among states that do use or encourage use of TBL, there 
was a significant difference in emphasis. Relative to their counterparts, states determined to have made 
institutional commitments generally reported a higher likelihood of TBL use or encouraged use in support 
of increasing the accessibility of WIA services for the eligible population. Among states reporting 
institutional commitments, 93 percent used or encouraged the use of TBL for increasing the accessibility 
of WIA services (versus 69 percent of those without institutional commitments). 


Exhibit 3.6: SWAs’ objectives advanced by use or encouraged use of TBL 


Percentage of | Percentage of 


states states not 
Statewide objectives for which the Percentage of reporting reporting 
SWA used or encouraged use of TBL (if state state institutional institutional 
used or encouraged use) respondents buy-in buy-in 
Increasing the accessibility of WIA services for 81 93+ 69" 


the eligible population (e.g., rural populations) 


Increasing individual options and personal 
flexibility in accessing WIA services (e.g., 80 87 73 
employed workers seeking services after hours) 


Improving cost efficiency in the WIA delivery 


system (e.g., limiting facilities costs) ue ee i 


Attracting priority target groups (e.g., younger 


populations, disabled populations) 70 64 75 


Promoting greater coordination of services 
across agencies (e.g., cross-program sharing of 68 67 69 
development costs for online courses) 


Notes: * Indicates that the difference between states that do and do not report institutional buy-in is statistically 
significant at the 10 percent level. 

All states, n=29-32 

States reporting institutional buy-in, n=14-16 

States not reporting institutional buy-in, n=14-16 

Only states that reported that their SWA used or encouraged use of TBL as a means to promote or complement 
existing objectives were asked about the statewide objectives. 

Additional detail provided in Exhibit A.3.6 in Appendix A 

Source: TBL state survey 


3.3.1 Use of resources to guide TBL 


SWA respondents reported on their SWA’s use of resources to guide TBL initiatives within their state’s 
workforce systems (Exhibit 3.7). Approximately one-third (31 percent) of respondents reported using 
digital content tools to support TBL, with fewer using digital libraries (20 percent), advisors on 
educational technology (15 percent), and groups of experts focused on educational technology (13 
percent). 


Respondents were given the opportunity to elaborate on their SWAs use of resources to guide TBL. One 
SWA respondent described her state’s use of a group focused on educational technology: 
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Our business advisory council meets on a quarterly basis to discuss program development that will 
meet institutional and student standards and performance.... [This also] includes a review of 
developing and current technology-based learning to ensure [we’re] keeping current with the latest 
technological business practices and industry standards. 


Exhibit 3.7: Use of resources to guide TBL 


Percentage of Percentage of 
states states not 
Percentage of reporting reporting 

state institutional institutional 
Resources used to guide TBL respondents buy-in buy-in 
Digital content tools to support TBL 31 31 30 
A digital library to support TBL 20 31 13 
An advisor on educational technology 15 33* 4* 
A group (e.g., commission, committee or 
taskforce of experts) focused on educational 13 19 8 
technology 


Notes: * Indicates that the difference between states that do and do not report institutional buy-in is statistically 
significant at the 10 percent level. 

All states, n=39-40 

States reporting institutional buy-in, n=15-16 

States not reporting institutional buy-in, n=23-24 

Percentages within columns do not sum to 100 since respondents could select multiple responses. 

Additional detail provided in Exhibit A.3.7 in Appendix A 

Source: TBL state survey 


Other administrators described their use of digital content tools in the context of their widespread 
applicability within the workforce system. For example: 


Currently, we use wikis for document sharing; Moodle-based courses to share resources and tools; 
and have a private LinkedIn group to share best practices for training customers on social media and 
job search skills. The [state] workforce system also has a Twitter account to share resources, 
information and training opportunities. 


The world of technology is no longer an isolated matter to the state workforce system. The use of 
Internet; audio and video tapes; and audio and video conferencing [demonstrates] where digital 
content is made available [and] are just some of the normal practices that are used to support 
technology-based learning. 


3.4 Statewide Availability of TBL Training Programs 


In addition to their insights into the nature of statewide support for 
technology-based learning, SWA administrators also were asked WIA-funded training services 
about the current availability of training programs offering within the median state 
technology-based skills building for training and education across e 1,548 eligible programs 
their state. Specifically, respondents were asked a series of e 166 eligible providers 
questions about the programs that were approved to receive WIA 
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funding. While localities may impose further criteria on approved training programs, a minimum 
requirement for using WIA funding for training activities is that the program be listed on the state ETPL. 
Therefore, understanding the availability of technology-based training and education among programs on 
the state ETPLs is a first step in gaining a broader understanding of its use across the workforce system. 


3.4.1 State criteria for TBL training programs 


Since the SWA plays an important role in developing and approving a state’s ETPL, the survey asked 
respondents if their state had specified any additional restrictions or requirements related to the use of 
technology-based training and education above and beyond specifications in WIA.” While WIA sets no 
criteria on the specific use of technology-based skills building for training services, nearly one in five 
respondents (19%) reported that their states have established criteria. Examples cited by survey 
respondents include: 


e No more than 50 percent of a course of study can be via TBL; 
e Only accredited universities and community colleges can use distance learning for WIA; and 


e Online coursework requires some face-to-face instruction by an instructor. 


Within these states, these types of specifications may serve to restrict the overall use of technology-based 
skills building at the local level. 


3.4.2. Availability of information about TBL in training programs 


While the use of technology-based skills building for occupational training has been encouraged within 
the state workforce system, states and LWIBs have leeway in determining whether and how to adopt 
these training models. In some cases, states may collect information about the prevalence of training 
programs with TBL by requiring local areas to report on the delivery mode of programs proposed for the 
state ETPL. SWA administrators were asked whether they gather information about the modes of 
delivery for the training programs on their ETPLs. Because state data systems varied in the level of detail 
on accessibility of training programs’ delivery modes, the question used the following definitions, with 
“distance learning” serving as a proxy for the use of technology-based skills building: 


e Distance programs are programs that are delivered online or through electronic linkages (e.g., 
videoconferencing). These programs may be entirely distance or they may be blended, offering 
courses or sessions both by distance and in person 


e In-person programs are programs that rely on face-to-face instruction (i.e., traditional programs). No 
course or session was held online or by distance. 


As shown in Exhibit 3.8, 65 percent of states had recorded information on the delivery mode for programs 
on their states’ ETPLs. These data potentially provide the opportunity to broadly examine the use of 
technology-based skills building among eligible training providers in these states. 


°° The Act specifies the types of allowable training activities and requirements for determining the eligibility of 


training providers. However, the Federal WIA statue and its regulations do not preclude eligible training 
providers from using TBL. Further TEGL 17-07 encouraged states and local areas to review their criteria for 
assessing training providers’ eligibility to ensure that “the use of TBL methods does not preclude an otherwise 
qualified training provider from eligibility” (U.S. DOL, 2008b, pp. 4-5) 
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Exhibit 3.8: Availability of information on mode of delivery for ETPL programs 


Not available, 
35% 


Available, 
65% 


Notes: n=40 
Additional detail provided in Exhibit A.3.8 in Appendix A 
Source: TBL state survey 


Within those states that tracked the use of technology-based skills building among eligible programs, 76 
percent reported the availability of any eligible offerings with technology-based skills building within 
their state (Exhibit 3.9). Further, in states that offered technology-based skills building, respondents 
indicated 9 percent of programs were delivered using distance or blended means.”’ It is important to note 
that, since these data are not uniformly available, these percentages can serve only as an estimate. 
Nonetheless, these data indicate that technology-based skills building opportunities for occupational 
training were available across most states. 


3° A 2006 poll indicated that, nationally, approximately 12 percent of organizations on states’ ETPLs provide 
distance learning opportunities for workers and job seekers. No information was available from that poll using 
programs as the unit of analysis (U.S. DOL 2006). 
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Exhibit 3.9: Prevalence of TBL for approved training programs in the median state, by presence of 
approved TBL program 


States with approved TBL programs States without approved 
(76% of states) TBL programs (24%) 
distance 
or 
blended, 
9% 


100% 


in- 
person, 
91% 


Notes: n=29. 

Size of each pie chart is approximately proportional to the percentage of programs for each provider type. 
Additional detail provided in Exhibit A.3.9 in Appendix A 

Source: TBL state survey 


Overall, these findings on the statewide availability of technology-based skills building for training 
services both characterize the availability of this TBL for training services and will serve as the backdrop 
for the AJCs’ use of technology-based skills building for training (see Chapter 6). 


3.5 Summary 


A majority of state respondents considered technology-based learning to be a high or medium priority 
within their state workforce systems. States that considered TBL a high priority offered examples of its 
benefits, including decreased time from unemployment to credential and employment, and the ability to 
meet increased demand for re-employment services, while states that considered TBL a medium, low, or 
not a priority tended to report that they considered TBL a local-level strategic issue or that other 
initiatives took priority over TBL. 


Additionally 44 percent of SWA respondents reported that their states made institutional commitments to 
TBL through their actions around legislation, policy, or funding decisions. States were focusing on TBL 
and establishing a supportive and structured context for local implementation. It is equally important to 
note that states that had not made such institutional commitments appeared to recognize the value of TBL 
and the complementary role it can play in advancing other statewide objectives. 


SWA responses also suggest that states generally had an infrastructure that was largely in place to track 
the extent to which technology-based skills building was used among eligible occupational training 
programs. Approximately 60 percent of all states that responded to the survey gathered sufficient 
information to monitor the mix of service delivery modes (including TBL) that were used in their 


3. The State Context for Technology-Based Learning pg. 23 


Exploring the Role and Adoption of Technology-Based Training and Employment Services 


approved training programs. Furthermore, technology-based skills building training opportunities were 
available in most (74 percent) of those states. In summary, while state commitments vary, SWAs were 
establishing the commitments and infrastructure needed to support continued use of TBL within the 


workforce system. 
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In addition to the state-level perspective summarized in the previous chapter, the research was also 
structured to gather insights from LWIAs. To this end, an online survey was conducted with the 
Executive Directors of LWIBs, who are broadly responsible for planning, designing, and implementing 
workforce development strategies at the local level. To complement the survey findings, a series of in- 
depth telephone interviews were conducted with nine LWIA respondents to gather additional detail on 
specific technology-based learning efforts that they referenced in completing their surveys.”* 


Where appropriate and feasible, this analysis examines differences in perspective among the LWIA 
respondents whose AJCs engaged with TBL for the greatest and least percentages of their participants. 
Specifically, the analysis examines two groups (see Exhibit 4.1): 


e LWIAs in which at least 90 percent of AJC participants engaged in TBL during PY 2011. These 
LWIAs represent the top 20 percent, or top quintile, of TBL participation. 


e LWIAs in which less than 5 percent of AJC participants engaged in TBL during that same year. 
These LWIAs represent the bottom 20 percent, or bottom quintile. 


While TBL offerings within AJCs may include both technology-based skills building and technology- 
based services, TBL participation in these contexts was likely skewed towards technology-based services. 
This was due to the fact that core and intensive services account for the largest proportion of WIA 
participants and these services, if they incorporate TBL, were more likely to do so in the form of 
technology-based services.” 


3! In their surveys, respondents were asked to describe any core or intensive services provided by their AJCs and 


which used “any cutting edge technologies or innovative TBL.” Eighty-eight LWIA representatives responded 
to this question. Their answers were analyzed and 18 respondents whose programs or practices appeared to use 
TBL in an innovative way were invited to provide additional information about their programs by e-mail or 
phone. Follow-up interviews were conducted with each of the nine LWIA participants who responded. 


°° See Chapter 5 for additional detail on technology-based services to promote core and intensive objectives 


within the workforce system 
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Exhibit 4.1: Distribution of reported TBL participation across LWIAs 


30 


20 


Percent of LWIAs 
10 


Bottom quintile 


0 10 20 30 40 50 60 70 80 90 100 
Percentage of LWIA Participants Who Took Part in Any TBL in PY 2011 


Notes: n=253 
Additional detail provided in Exhibit A.4.1 in Appendix A 
Source: TBL local survey 


To indicate whether differences are meaningful or may be due to chance, the analytic exhibits include an 
asterisk if the difference is statistically significant at the 10 percent level based on the results of t-tests 
when comparing two proportions and on chi-squared tests when comparing sets of categories.** For 
example, LWIAs in states that reported having made institutional commitments to TBL had significantly 
more of their participants engaged in TBL than LWIAs in states without such commitments.** While this 
categorization of LWIAs into the top and bottom quintiles is based on respondents’ estimates, those 
estimates are broadly indicative of the spread of TBL use throughout their service populations. As such, 
these subsamples of LWIA respondents will be used throughout the analysis as a way to identify and 
compare the most and least active users of TBL. 


This chapter begins with a discussion of the resources and services offered within the American Job 
Centers during PY 2011. Against this broad landscape of services, the research then examines a number 
of ways LWIAs may have supported the integration of technology-based skills building and technology- 
based services into their service delivery. The areas of investigation include marketing and promotion of 
TBL, investment in and enabling access to equipment and infrastructure, and the provision of participant 
support in the use of technology. 


3 See Chapter 2 for additional information on the study’s methodology. 


4 See Appendix A for additional detail. 
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4.1 Use of Technology to Deliver Services through American Job Centers 


AJCs provide central access points within each LWIA for a range of job search, placement, training, and 
education services. While these services are traditionally provided in-person at an AJC location, services 
may now additionally be accessed using electronic services, either through: 


e Virtual AJCs (e.g., Virtual One-Stop), or 


e Electronic linkages to (physical) AJCs, which can also be accessed in person. 


Just under one-fifth (19 percent) of LWIBs surveyed contracted with one or more virtual AJCs to provide 
services (Exhibit 4.2). 


Exhibit 4.2: Prevalence of virtual AJCs across LWIAs 


On-line only 
AJCs, 19% 


No on-line only 
AJCs, 81% 


Notes: n=405 
Additional detail provided in Exhibit A.4.2 in Appendix A 
Source: TBL local survey 
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Additionally, it should be noted that the availability of virtual AJCs did not vary by TBL participation 
across LWIAs. These virtual AJCs were equally distributed across the LWIA landscape and thus do not 
appear to have contributed to variation in AJC participants’ use of TBL. 


TBL may also be incorporated into the services provided at (physical) AJCs. Exhibit 4.3 shows that one- 
third (33 percent) of LWIA respondents reported that their AJCs offered some services using only 
electronic technology (e.g., virtual desktops, e-mail distribution lists, videoconferencing). Additionally, 
most LWIA respondents (79 percent) reported that, in addition to in-person services, AJCs offered 
blended services that combine in-person access and electronic technology use. Later chapters describe 
how electronic technology was used in a range of workforce services, including assessment, career 
readiness, job search, training, and education. 


Exhibit 4.3: Delivery modes employed by (physical) AJCs 


100% 
100% 
80% | 
60% 
79% 
80% 40% | 
2 20% - 
5 62% 
9 60% - 0% : 
o Electronic Blended  In-person 
< 
= 2 90% TBL participation 
2 40% 
3 33% 100% - 
2 
80% | 
20% ~ 60% 
40% | 
0% 20% - 
Electronic Blended In-person 0% 
AJCs’ delivery mode for core and intensive services Electronic Blended*  In-person 
All LWIA Respondents < 5% TBL participation 


Notes: * Indicates that the difference between subgroup and remainder of sample is statistically significant at the 10 
percent level. 

All LWIAs, n=431. 

2 90 % TBL participation, n=47 

< 5% TBL participation, n=52 

Only respondents who reported that their LWIA had one or more Comprehensive American Job Center completed 
this question. 

Percentages across rows do not sum to 100 since respondents could select more than one type of delivery mode. 
Additional detail provided in Exhibit A.4.3 in Appendix A 

Source: TBL local survey 
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Differences in levels of TBL participation across AJCs appear to be based in the prevalence of 
technology-based skills building and technology-based services offerings within physical AJCs (Exhibit 
4.3). Specifically, LWIAs with less than 5 percent of AJC participants engaged in TBL (i.e., the bottom 
quintile) were significantly less likely than others to have blended offerings. That is, not only did LWIAs 
in the bottom quintile have low levels of TBL participation, they also had fewer TBL options within their 
AJCs. While this may appear to be an intuitively obvious relationship (fewer offerings equates to lesser 
participation), the reverse does not seem to hold. Specifically, the data indicate that there were not 
significant differences between LWIAs in the top quintile of TBL participation and other LWIAs. While 
this latter finding may be partially attributable to sample sizes, it may also suggest that LWIAs with 
greater than 5 percent TBL participation already had some of the infrastructure in place within their AJCs 
to increase their TBL participation. In addition, it implies that a TBL strategy may require resource 
availability combined with proactive efforts to engage AJC participants. This important distinction is 
discussed in greater detail below. 


4.2 Supports LWIAs Offered Participants for TBL 


In considering the use of TBL at the local level, it is important to make the distinction between provision 
and engagement. In other words, AJCs may offer a variety of technology-based skills building and 
technology-based services, but there may be other factors at play that impede participants’ full 
engagement with those services. This may include, for instance, prospective participants’ levels of 
technological literacy, alignment of TBL opportunities with service needs, and access to the necessary 
equipment or technology. This section investigates these types of efforts to engage participants with 
TBL. 


4.2.1 LWIAs’ strategies to promote TBL 


Promotion is a particularly critical dimension of the TBL strategy since AJCs may seek to establish 
interest and credibility among a whole host of stakeholders including participants, providers, and 
businesses. LWIA respondents emphasized the importance of promoting TBL to participants as a step in 
engaging them with the technology. As shown in Exhibit 4.4, more than 70 percent of LWIA respondents 
reported that their AJCs promoted TBL to their participants. Moreover, emphasis on promotion varied 
significantly by levels of TBL participation. In particular, the respondents in the highest quintile of TBL 
participation were significantly more likely to promote TBL (87 percent). 


Exhibit 4.4: Prevalence of promotion of TBL to AJC participants 


Percentage of Percentage of 
Percentage of LWIAs with LWIAs with 


LWIA 290% TBL <5% TBL 
Was TBL promoted? respondents Participation Participation 
Promoted TBL to AJC participants 72 87* 46* 


Notes: All LWIAs, n=425. 

2 90 % TBL participation, n=46. 

< 5% TBL participation, n=52. 

* Indicates that the difference between subgroup and remainder of sample is statistically significant at the 10 percent 
level. 

Additional detail provided in Exhibit A.4.4 in Appendix A 

Source: TBL local survey 
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AJCs used a variety of means to promote TBL (Exhibit 4.5). Among those engaged in promotion, nearly 
all (98 percent of those promoting TBL) relied on word of mouth, for example, through case manager 
recommendations. Other common forms of promotion included using training sessions and handouts (74 
and 72 percent, respectively). Additionally, many respondents reported using technology-based means or 
media to promote TBL. Over half of the respondents reported using some form of online promotional 
materials (63 percent) or social media promotion (54 percent) to advance their TBL agenda. Finally, the 
use of electronic and online promotions (63 percent) appears to be favored over traditional media 
marketing modes (38 percent). 


Exhibit 4.5: Means of promoting TBL 


Word of mouth osx 
Training session [NN 74. 
Printed or electronic handouts Eee) 72% 
Online promotional materials aa 63% 
Social media promotion [NN sax 


Traditional media advertising | 38% 


AJCs’ means of promoting TBL 


0% 20% 40% 60% 80% 100% 
Percent of LWIA respondents 


Notes: n=291-300 

Percentages across rows do not sum to 100 since respondents could select more than one type of promotion. 
Additional detail provided in Exhibit A.4.5 in Appendix A 

Source: TBL local survey 


4.2.2 Equipment or technology used by AJC participants 


While TBL delivery options have the capacity to increase cost efficiency, they often require up-front 
investments in equipment and/or infrastructure. The data regarding these investments should be reviewed 
in the context of a resource-constrained delivery system that may prioritize services to a lower-income 
population who may not have a steady source of income or many assets. 


Respondents reported that AJC participants had access to basic computer and electronic technology that 
might be used for technology-based skills building, technology-based services, or both. Most LWIA 
respondents reported that their AJCs provided computers for their participants’ use (95 percent; Exhibit 
4.6). Moreover, these respondents reported that they could rely on participants to have some access to 
computer equipment outside of the formal delivery system. This was most commonly available through 
other public sources, such as libraries (92 percent), or personal equipment (85 percent). 
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Exhibit 4.6: Reported source of equipment or technologies used by TBL participants 


Public (e.g., library) as | 92% 
AJC participants ee 85% 
AJC participants’ employers | 58% 


0% 20% 40% 60% 80% 100% 
Percent of LWIA respondents 


AJC participants’ source of equipment or technology 


Notes: n=221-391 
Additional detail provided in Exhibit A.4.6 in Appendix A 
Source: TBL local survey 


In addition to basic computer access, the survey shows that AJCs had made considerable investments in 
technology that improves the access of their participants (Exhibit 4.7). Nearly all offered some type of 
on-site Internet access (94 percent) and more than half offered access to video or audio devices to their 
participants (63 and 54 percent, respectively). The data also indicates that fewer investments were made 
in technologies that were either still emerging for TBL use (mobile devices, 9 percent of respondents) or 
that required some level of critical mass to fully justify widespread availability (virtual desktop,’” 31 
percent of respondents). It is important to point out that the investments in remote access (e.g., virtual 
desktops) appear to have gained considerable traction among those LWIAs in the top quintile of TBL 
participation, where half of LWIA respondents (51 percent) reported that their AJCs made this technology 
available. 


© Virtual desktops may allow participants to access remotely software licensed by the AJCs. 
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Exhibit 4.7: Equipment or technologies made available to AJC participants 


Percentage of Percentage of 
Percentage of LWIAs with LWIAs with 
LWIA 290% TBL <5% TBL 

Equipment or technology respondents participation participation 
On-site computer or on-site laptop or 95 98 78* 
computer lab 
On-site Internet access 94 100* 80* 
Video device (e.g., DVD player, television) 63 85* 37* 
Audio device 54 79% 28% 
Tele-conferencing or videoconferencing 36 46 16* 
equipment 
Virtual desktop or remote access 31 5i* 10* 
Loaned computer or loaned laptop 11 20* 4* 
Mobile device 9 16 2* 


Notes: * Indicates the difference between subgroup and remainder of sample is statistically significant at the 10 
percent level. 

All LWIAs, n=362-390 

2 90 % TBL participation, n=43-46. 

< 5% TBL participation, n=50-52. 

Additional detail provided in Exhibit A.4.7 in Appendix A 

Source: TBL local survey 


4.2.3 Activities supporting participants’ use of technology 


A commonly cited barrier to TBL use in general was participants’ low levels of technological literacy or 
comfort (see, for example, TEGL 17-07). This can be a particularly relevant concern given both the 
actual pace of technological change and the accompanying perception (warranted or not) that participants’ 
skill levels may not be sufficiently current. LWIA respondents were asked whether their AJCs used any 
specific intake or orientation activities to gauge their participants’ readiness to pursue services using TBL 
(Exhibit 4.8). Nearly two-thirds of respondents (65 percent) reported that their AJCs provided an 
assessment or interview to determine their participants’ technology readiness. Fewer respondents (48 
percent) reported their AJCs used assessments or interviews related to course hardware, software, or 
equipment requirements. 


Beyond the initial assessment, respondents reported that their AJCs provided some degree of proactive 
intervention to enhance participants’ comfort and skill level in using the learning technology. 
Approximately two-thirds (66 percent) reported that technical assistance was offered, while one-third (37 
percent) reported that AJCs offered a formal orientation to course technology or to a learning 
Management system. 
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Exhibit 4.8: Activities supporting the use of technology 


Percentage of | Percentage of 
Percentage of LWIAs with LWIAs with 
LWIA 290% TBL <5% TBL 

Activities to support use of technology respondents participation participation 
Assessment activities 
Assessment or interview to assess technology 65 78 38* 
readiness 
Assessment or interview related to course 48 53* 34* 
hardware, software, or equipment requirements 
Support activities 
Technical assistance to support participants’ use 66 82* 33* 
of TBL 
Formal orientation to course technology or to 37 47 27* 
learning management system 


Notes: * Difference between subgroup and remainder of sample is statistically significant at the 10 percent level. 

All LWIAs, n=349-372. 

2 90 % TBL participation, n=21-36. 

< 5% TBL participation, n=13-19. 

Additional detail provided in Exhibit A.4.8 in Appendix A 
Source: TBL local survey 


The data in Exhibit 4.8 shows a comparison in the levels of support activities across the top and bottom 
quintiles of LWIAs defined by their percentage of AJC participants using TBL. Among LWIAs with at 
least 90 percent of AJC participants engaging in TBL, the vast majority (82 percent) reported investing in 
technical assistance. Likewise, LWIAs with low levels of TBL participation were significantly less likely 
than others to include any of these support activities (e.g., 33 percent reported investments in technical 
assistance). The message across these findings emphasizes the idea that these activities not only address 
the technological skills and the comfort level of participants, but also may actually serve to attract interest 
in technology-based skills building or other technology-based services. Further, engagement in these 
support activities may complement efforts to engage more participants in TBL and build technology 
literacy among participants. 


4.3 Summary 


Information from LWIA respondents described a profile of efforts to integrate TBL into local service 
offerings and delivery systems. Seventy percent of LWIA respondents reported their AJCs offered 
participants some services through a blended mode; an additional 30 percent offered some form of 
exclusively electronic services. An additional one in five of the LWIAs offered services through virtual 
AJCs. 


LWIA respondents recognized the importance of promoting TBL use, with 70 percent reporting that they 
engage in some form of proactive outreach to participants. Nearly all LWIA respondents (upwards of 90 
percent) reported that their AJCs provided participants with foundational equipment for TBL, computers 
and Internet access. Further, 66 percent of respondents reported that their AJCs were supporting 
participants’ use of technology through targeted technical assistance. 
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Efforts by AJCs to engage participants in either technology-based skills building or technology-based 
services may contribute to greater utilization of TBL. At the most active end of the spectrum, LWIA 
respondents reported nearly 90 percent of participants engaged in TBL—this represents 20 percent of 
LWIAs, or the top quintile. Their counterparts at the least active end of the spectrum (bottom quintile of 
LWIAs) reported TBL participation levels of less than 5 percent. These two categories of LWIAs 
differed, not only in their levels of TBL participation but also in the contexts they established for 
technology-based skills building and technology-based services. LWIAs in the top quintile were more 
likely than others to report their AJCs used blended delivery modes for services, promoted TBL usage, 
provided participants with access to technology, or supported participants’ use of technology through 
technical assistance. 
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This chapter reviews the use of TBL to support the delivery of WIA Title I core and intensive services. 
Core services involve the provision of information on job openings, the labor market, and providers of 
training services, youth activities, adult education, vocational rehabilitation activities, and vocational 
education. Intensive services involve individualized activities such as counseling and assessment to help 
customers choose training programs and select occupational areas. Core services are available universally 
to any adult or dislocated worker within the local delivery system; intensive services are targeted and 
available to a subset of participants deemed to need additional support to obtain or retain employment.”° 


TBL may be incorporated into core and intensive services in a number of ways. The majority of 
activities can best be characterized as having the potential for incorporating technology-based services, 
including: 
Searching computerized or online databases to obtain labor market information, conduct career 
exploration, or search for jobs; 
Connecting with employers and other job seekers via social media and social networking sites; and 
Accessing presentations and webinars by career speakers asynchronously through digitally archived 
recordings. 


However, technology-based training and education can be used in support of core and intensive services 
to build specific skills, such as: 


e Practicing interviewing skills using avatars in a virtual work setting and 
e Learning job application skills through web-based resume writing tutorials. 


While WIA core and intensive services represent distinct types of services, this study presents them in an 
integrated discussion. Collectively, core and intensive services are typically provided directly by AJCs, 
under contract and oversight from LWIBs. As such, their integration of TBL was likely shaped by 
common factors within the LWIA. This stands in contrast to the provision of training under WIA (see 
Chapter 6) or Title II adult education services (see Chapter 7), where use of technology-based skills 
building is more likely to be influenced by external providers. 


The chapter includes a review of service delivery methods and platforms used to access services during 
PY 2011. Where possible, it provides details on core and intensive services separately and together. 
These issues will be explored for the universe of LWIAs while making comparisons, as appropriate, to 
those with at least 90 percent of AJC participants engaged in TBL and those with less than 5 percent of 
AJC participants engaged in TBL (i.e., the top and bottom quintiles of TBL participation, respectively; 
see chapter 4). To indicate if these differences are meaningful or may be due to chance, the analytic 
exhibits include an asterisk if the difference is statistically significant at the 10 percent level based on the 
results of t-tests when comparing two proportions and on chi-squared tests when comparing sets of 
categories. These subsamples of LWIA respondents serve as a proxy for identifying and comparing the 
most and least active users of TBL. 


36 Adults and dislocated workers who are unemployed and unable to obtain employment through core services, 


who are in need of more intensive services to obtain employment, or who are employed and in need of more 
intensive services in order to obtain or retain employment that allows for self-sufficiency (WIA section 134). 
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5.1 Delivery Mode for Specific Core and Intensive Services 


To determine the extent to which TBL was being incorporated into the delivery of core and intensive 
services, LWIA respondents were asked how these services were delivered within their local delivery 
system. They could describe their AJCs’ services as: 


e Electronic technology services: participants could access services only through electronic technology 
(e.g., online, in a computer lab, videoconferencing); 


e Blended services: participants could access services both through electronic technology and in person; 
or 


e In-person services: participants could access services only by physically visiting the local delivery 
system (e.g., in-person, one-on-one intake or counseling; a traditional classroom-based course). 


Within this framework, TBL encompasses the categories of electronic technology services and blended 
services. 


Exhibit 5.1 summarizes the survey findings for the delivery of core services. In reviewing these data, it is 
important to remember that, in this context, the term “learning” (in technology-based learning) must be 
applied broadly. That is, the delivery of core services is often about information access and information 
sharing to promote solid planning and decision-making by participants. As such, the use of TBL for these 
services likely refers to technology-based services. 


Three aspects of Exhibit 5.1 are noteworthy: 


e Active use of electronic (only) technology: The Department made investments in an array of online 
resources that support the delivery of core services. The baseline level of services delivered via 
electronic (only) technology-based services likely reflects active use of such resources as the 
Healthcare Virtual Career Network (www.vcn.org) for career exploration and training information 
related to health care careers, and My Skills My Future (www.MySkillsMyFuture.org) for matching 
occupational experiences with skills needed in other occupations.” 


e TBL for information sharing: Many core services centered on information transfer rely on a 
blended approach, combining both in-person and electronic technology services. This trend in 
technology-based services for knowledge building holds for information flowing both to and from 
participants. Approximately three-quarters of respondents reported that their AJCs integrated a 
blended component in the labor exchange and job search process (e.g., labor market and career 
information, 80 percent; and job search and placement, 78 percent). Further, about a quarter of 
respondents reported these services were available in their LWIAs using only electronic technology 
(labor market and career information, 26; and job search and placement, 22 percent). 


7A complete listing of online resources made available by DOL is included in Training and Employment Notice 


Number 29-12 at http://wdr.doleta.gov/directives/attach/TEN/TEN_29 12.pdf. 


5. Technology-Based Learning for WIA Core and Intensive Services pg. 36 


Exploring the Role and Adoption of Technology-Based Training and Employment Services 


Exhibit 5.1: Delivery mode for core services provided by AJCs 


Information transfer L 
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Percent of LWIA respondents 


Notes: n=396-421 

Only respondents who reported that their LWIA provided these types of services completed these questions. 
Percentages across rows do not sum to 100 since respondents could select more than one type of delivery mode for 
each category of services provided. 

Additional detail provided in Exhibit A.5.1 in Appendix A 

Source: TBL local survey 


e Commitment to in-person services: Despite the prevalence of blended options and the presence of 
exclusively electronic options, many LWIA respondents reported that their AJCs were providing 
exclusively in-person services. Their commitment to providing-in-person services is apparent in 
bringing participants into the system (outreach intake and orientation, 67 percent). Similarly, LWIA 
respondents reported their AJCs’ commitment to in-person support for the key knowledge transfer 
activities discussed above, such as the use of labor market information (45 percent) and exploration of 
career, training, and support service options (78 percent). 


Collectively, the data indicate that technology-based services, particularly blended approaches, have been 
integrated into the delivery of core services. At the same time, the use of in-person delivery options for 
some services suggests that in-person service delivery remains important and/or that technology-based 
services approaches are not currently viable. 


Exhibit 5.2 provides an overview of the methods used in the delivery of intensive services that were 
offered within AJCs. In contrast to core services, intensive services may incorporate TBL through both 
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technology-based services and technology-based skills building. Overall, this profile suggests that, 
relative to core services, these activities were less likely to have incorporated any type of electronic 
technology into their delivery. Several points are noteworthy: 


e Technology-based skills building for developing foundational skills: Those intensive services 
intended to develop foundational skills (e.g., job application skills) had established a level of blended 
delivery among 40 to 50 percent of the LWIAs that responded. The lowest prevalence of technology- 
based skills building for foundational skills was in the delivery of soft skills (37 percent), which were 
more likely to rely on interpersonal interaction and instruction. 


The Workforce Alliance of South Central Kansas’ most popular classes are a series of blended 
courses to build Microsoft Office skills. Through online content and in-person instruction, participants 


build technological literacy. Classes are kept relevant as instructors incorporate new developments in 
workforce trends and work with Microsoft specialists who assess utility and certifications for the classes. 
Businesses have contracted with the LWIB to provide sessions for employees. 


e Technology-based services for assessment: Relative to other intensive services, the assessment 
functions were more likely to be delivered using a blended approach. Similar to some of the core 
services, assessment can be characterized as a form of information transfer or knowledge building. 
Perhaps not surprisingly, technology-based services were reported to be more prevalent in support of 
diagnostic testing (53 percent blended; 20 percent electronic technology only) than in support of 
assessments designed to identify individual employment goals and barriers (35 percent blended; 12 
percent electronic technology only). 


e In-person counseling and case management: Counseling and case management services had a low 
use of electronic technology and were dominated by in-person delivery options. However, the 
finding that blended delivery options for counseling and case management services were reported by 
between 10 and 20 percent of the respondents indicates that there is potential for these services to be 
delivered using blended approaches. 


While these general trends in intensive service delivery hold for LWIAs at the top and bottom of the TBL 
participation distributions, there were some significant differences between the groups (see Appendix A). 
Generally, the delivery modes used by the top quintile of LWIAs were not very different from those used 
by other LWIAs. This suggests that the current trends for incorporating TBL into many of these services 
may have been leveling out since the most active users tend to mirror the broader average. There are 
some notable exceptions. LWIAs with at least 90 percent of participants engaged in TBL were more 
likely than others to use technology-based services for bringing participants into the system (66 percent 
blended and 26 percent electronic only). These most active LWIAS were also more likely to report using 
electronic-only job search groups, though such users were a minority (14 percent). 
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Exhibit 5.2: Delivery mode for intensive services provided by local delivery systems 


Foundational skills 


SSS SSeS H G0, 
Computer software skills ee conic y 

P Blended, A ieoeeon, 66% 

9, 
Computer skills ey 43% 
ie) 
Job application skills es 44% 
ie) 
soft skills Miiiiahddies 37% 


69% 
72% 


77% 


Assessment 


9 
Diganostic testing eT eo 53% 6 
Erp tbarriers & I ies 
mployment barriers 36% 


goals 76% 


5% 


Counseling & 


case management 9 
Job search groups esas soe 22% 
9 
Seeking training services Mito 19% 


Individual counseling & aan 326 baa 
career planning : aie 


Individual employment mm S% 16% 
plans ai 


Intensive services provided by AJCs 


85% 


86% 


0, 
Group counselin m3% 9 
P 8 2 94% 


0% 20% 40% 60% 80% 100% 
Percent of LWIA respondents 


Notes: n=278-423 

Only respondents who reported that their LWIA provided these types of services completed these questions. 
Percentages across rows do not sum to 100 since respondents could select more than one type of delivery mode for 
each category of services provided. 

Additional detail provided in Exhibit A.5.2 in Appendix A 

Source: TBL local survey 
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At the other end of the distribution, LWIAs in the bottom quintile of TBL participation tended to be less 
likely than others to report using blended delivery for most core and intensive services. These LWIAs 
were also less likely to deliver intensive services using only electronic means. The exception to the lesser 
use of technology-based services among bottom quintile LWIAs was counseling or case management 
services, where LWIAs at all participation levels predominantly used in-person services. 


5.2 Use of TBL for Core and Intensive Services 


The previous section introduced the various core and intensive services with an emphasis on the 
predominant delivery mode(s) by which they were offered to participants. The discussion that follows 
looks more closely at the TBL component itself in order to characterize the use of technology. As 
illustrated above, TBL for core and intensive services may primarily be considered technology-based 
services, but may also include technology-based skills building. To understand TBL usage by AJCs, the 
discussion reviews the scheduling modes for core and intensive services. It also examines the 
predominant platforms being used to access core and intensive services that use TBL. This section 
examines only relevant responses (i.e., respondents reporting the use of either electronic or blended 
delivery), and combines core and intensive services to increase sample sizes and analytic scope. 


5.2.1 Scheduling modes for core or intensive services 


One feature of TBL is that it can allow for asynchronous learning, that is, learning at each participant’s 
individual pace and schedule. In contrast, synchronous learning, as occurs in a traditional classroom, 
requires that both the instructor and the learner be in the same (physical or virtual) place at the same time. 


Through technology, core and intensive services can be offered either asynchronously, synchronously, or 
using a combination of both scheduling modes. Exhibit 5.3 summarizes respondents’ characterization of 
the scheduling modes for core and intensive services in their AJCs. While the vast majority of 
respondents reported their AJCs offered these services on a fixed schedule (89 percent), the near- 
universal reported use of combined (synchronous and asynchronous) scheduling suggests that AJCs were 
seeking a higher degree of flexibility. Finally, somewhat fewer AJCs offered any core and intensive 
services on a strictly asynchronous schedule (79 percent). It is important to note that this percentage was 
significantly lower among LWIAs in the bottom quintile of TBL participation (70 percent), suggesting 
that a number of these AJC may be able increase their use of asynchronous delivery and add choices for 
participants. 


AJCs in Nashville (TN) are using technology to better prepare job seekers for employment interviews. 
These job application skills are taught both in person and using blended delivery modes combining 
synchronous and asynchronous scheduling. In addition to providing participants with interviewing 
techniques, guidance, and practice in an in-person workshop, local service providers also videotape 


practice interviews and provide opportunities for self-review under the guidance of a career coach at a 
later point in time. Other local delivery systems subscribe to websites, such as InterviewStream or 
Perfect Interview, that provide interview question banks, model good interviews, and record practice 
interviews for sharing with coaches asynchronously. 
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Exhibit 5.3: Scheduling modes for core or intensive services 


Percentage — Percentage 


of LWIAs of LWIAs 
Percentage with 290% with <5% 
of LWIA TBL TBL 
Scheduling modes respondents | participation participation 


Asynchronous: Services occurred individually, at the 
participants’ own pace (i.e., there were no scheduled 
class sessions) 

Combined: Services occurred both during scheduled 
sessions and individually, at the participants’ own pace 
Synchronous: Services occurred at a scheduled time 
and location (in either a physical or virtual classroom) 


89 85 92 


Notes: * Indicates that the difference between subgroup and remainder of sample is statistically significant at the 10 
percent level. 

All LWIAs, n=415-420. 

2 90 % TBL participation, n=46. 

< 5% TBL participation, n=50-52. 

Percentages across rows do not sum to 100 since respondents could select more than one type of scheduling mode 
for their core and intensive services. 

Additional detail provided in Exhibit A.5.3 in Appendix A 

Source: TBL local survey 


5.2.2. Platforms used for core or intensive services 


In this research, TBL, by definition, encompasses a wide variety of technology-based media, platforms, 
and/or communication devices and modes for either service provision or skills building. Exhibit 5.4 
indicates that the personal computer (laptop or desktop) was the most common platform used to access 
any core or intensive services. At the same time, the data suggest that smartphones, as well as various 
video players and videoconferencing equipment, may represent options for accessing these workforce 
development services. These platforms were reported by approximately half the LWIA respondents (50, 
53, and 43 percent, respectively). Less common was the use of traditional broadcast media (radio and 
television) or audio players for communicating information related to core and intensive services. 


The Lower Rio Grande Valley WIB (TX) has integrated iPad tablets into its core and intensive 
services. The Director of Performance and Innovation has designed intuitive apps to match job seekers 
to jobs listed on the state workforce commission’s website (WorkInTexas.com), on the basis of their 
skills and interests and apps that capitalize on user’s location information to display nearby job 
openings. iPads were used to engage participants who have low technological literacy with electronic 


content and to efficiently use staff time and resources. That is, the user-friendly, intuitive interfaces of the 
iPads, combined with participants’ familiarity with touch-screen tablet computing (e.g., from using their 
personal smart phones), provides participants a way to access content with minimal staff guidance. 
Finally, via the FaceTime app, the local delivery system uses iPads as a low-cost alternative to 
traditional videoconferencing. 
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Exhibit 5.4: Media used to access core or intensive services 


Percentage 
of LWIAs Percentage 
Percentage with 290% of LWIAs 

of LWIA TBL with <5% TBL 
Media respondents | participation participation 
Computer (laptop or desktop ) 97 100 90* 
Telephone (audio only) 74 70 63 
Video player (e.g., VCR, DVD player) 53 62 35* 
Smartphone (e.g., Android phone, iPhone) or Tablet 50 62% 31% 
computer (e.g., iPad, Amazon Kindle Fire) 
Videoconference equipment 43 53* 16* 
Television 34 38 28 
Audio player (e.g., CD player, iPod) 22 30 16 
Radio " 12 14 6 


Notes: * indicates that the difference between subgroup and remainder of sample is statistically significant at the 10 
percent level. 
All LWIAs, n=389-417 
2 90 % TBL participation, n=42-47 
< 5% TBL participation, n=48-52 
Additional detail provided in Exhibit A.5.4 in Appendix A 
Source: TBL local survey 


LWIAs in the bottom quintile of TBL participation were less likely than their counterparts to report their 
AJCs enabled participants to use a wide variety of platforms to access core or intensive services. LWIAs 
in which fewer than 5 percent of AJC participants engaged in TBL lagged other LWIAs in the use of all 
platforms except for the most infrequent types: television, audio players, and radio (28, 16, and 6 percent, 
respectively across LWIAs). The exceptions may reflect the relatively smaller roles of these platforms 
more broadly. 


5.2.3. Communication mode for core or intensive services 


Finally, AJCs used a variety of technology-based communication modes to deliver core or intensive 
services (see Exhibit 5.5). The data indicate that more broadly well-established technologies were also 
prevalent in workforce development. For instance, LWIA respondents reported that phone calls and e- 
mails were used by virtually all AJCs (94 and 93 percent, respectively). In addition, a substantial 
majority of LWIA respondents reported their AJCs (72 percent) used some form of electronic documents 
(e.g., online resource materials, e-books) and nearly two-thirds (63 percent) reported their AJCs used 
social networking sites in order to support the communication for core and intensive services. It is 
particularly important to point out that the top quintile of TBL users did not exceed other LWIAs, 
suggesting that these were broadly pervasive trends. 


Equally noteworthy is the use of a broad range of communication modes extending beyond the common 
options previously discussed. The use of webinars and videoconferencing, particularly by LWIAs in the 
top quintile of TBL participation (43 and 28 percent, respectively, overall, and 62 and 44 percent, 
respectively, in the top quintile) clearly demonstrated efforts to establish greater interactivity. Similarly, 
the use of text messaging and conference calls, cited by 4 out of 10 LWIA respondents, underscores the 
important role being played by mobile phones in the delivery of core and intensive services. Even some 
of the more cutting-edge technologies (e.g., simulations, virtual classrooms) were reported to be in use by 
10 percent of LWIAs. 
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Exhibit 5.5: Communication modes used to provide core or intensive services 


Percentage of Percentage of 
Percentage of LWIAs with LWIAs with 
LWIA 290% TBL <5% TBL 

Communication Mode respondents participation participation 
Phone call (person-to-person) 94 100* 83* 
E-mail 93 96 81* 
Electronic documents (e.g., CD-ROM, online 72 93+ 43 
documents, e-books ) 
Social networking sites 63 63 39* 
Video files (e.g., non-interactive television broadcast 55 7A 57% 
DVD, YouTube, webcast) 
Webinar (i.e., live online conference or seminar) 43 62* 42* 
Conference call (voice only) 42 56* 12* 
Text messaging (by phone) 39 41 16* 
Videoconference 28 44* 4* 
Online messaging (i.e., instant messaging or IM) 21 25 4* 
Audio files (e.g., non-interactive radio broadcast 

. 20 29 12* 
compact disc, mp3 file, podcast) 
Online discussion board or message board 18 17 8* 
Live online discussion (e.g., chat room) 14 20 2* 
Digital or electronic games or simulations 13 15 6 
Virtual classrooms (e.g., Second Life) 10 16 2* 
Online collaborative workspaces (e.g., wikis, course 

7 7 0* 

blogs) 
Interactive television broadcast (e.g., call-in 6 5 5 
television program) 
Interactive radio broadcast (e.g., call-in radio 5 gt o* 
program) 


Notes: * Indicates that the difference between subgroup and remainder of sample is statistically significant at the 10 
percent level. 

All LWIAs n=389-393 

2 90 % TBL participation, n=43-47 

< 5% TBL participation, n=49-52 

Additional detail provided in Exhibit A.5.5 in Appendix A 
Source: TBL local survey 


A number of LWIBs cited the growing role social media play in their local delivery systems. The 
Mohave-La Paz LWIB (AZ) uses Facebook to connect employers and participants, to promote 


program activities for hard-to-reach groups (e.g., youth), and to inform participants of employment 
opportunities. This work is supported by a social media helpdesk, which answers general questions that 
users post on the site. 


5.3. Summary 


The data in this chapter show TBL—particularly technology-based services—was a strategy for 
delivering core and intensive services. All types of core and intensive services have the potential for 
being delivered, at least in part, electronically, as demonstrated by the fact that each of the services was 
delivered using technology-based services or technology-based skills building in at least some AJCs. 
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However, there were significant variations in which services were more or less likely to be delivered 
using TBL. AJCs were more likely to use TBL for core services that entail information sharing and 
assessment. Case management and counseling services were predominantly in-person. Across core and 
intensive services, technology-based services and technology-based skills building were being provided in 
a variety of ways: through asynchronous and synchronous approaches and using a variety of media and 
communication modes, including smartphones, as well as various video- and videoconferencing 
equipment. 
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Under WIA, AJC participants who are eligible for training services obtain them through a voucher-based 
system. Using ITAs, participants effectively purchase their training from a pre-approved set of eligible 
programs. Eligible training programs may include occupational skills training, on-the-job training, 
combinations of workplace training and related instruction, private sector training, skill upgrading and 
retraining, entrepreneurial training, job-readiness training, or other customized training. ETA has 
promoted TBL, particularly technology-based skills building to increase access to training for a larger 
number of participants, particularly those who cannot attend traditional classroom training courses.”® 


This chapter examines the extent to which technology-based skills building was incorporated into the 
training services in which participants enroll. While an LWIA may have had an extensive array of 
approved programs from which to select (median 135 programs at 25 providers per LWIA), there were 
typically a smaller number of programs that had actually enrolled training participants with ITAs. To 
ensure consistency, survey respondents were asked to focus on the programs serving the highest volume 


of AJC participants. Specifically, LWIB Executive Directors were 


asked to provide information on the five training programs in their WIA-funded training-PY 
LWIA that served the highest number of participants using WIA 2011 medians (self-reported) 
funds, such as ITAs, during PY 2011. Throughout this chapter, it 339 participants in PY 2011 
is important to note that the unit of analysis is the training program, 135 approved programs 


that is, the session, course, or course of study delivered by a given 25 approved providers 


provider.*” 


This chapter begins by examining the criteria LWIBs established for determining training program 
eligibility with respect to technology use. It then turns to documenting the use of technology-based skills 
building for training services and characterizing the types of programs that use technology-based skills 
building, focusing on the types of service providers, industry sectors, and credentials associated with 
those programs. 


38 


39 


40 


U.S. DOL, 2008b 


Pre-testing suggested that LWIA respondents varied in their ability to provide information in delivery mode for 
all training programs within the LWIA because this information was not systematically recorded across LWIAs 
at the time of the survey. However, in 2012, ETA stated its intention to systematically collect, through the WIA 
Standardized Record Data (WIASRD) reporting system, data on whether training participants received training 
through distance learning (U.S. DOL, 2012b). 


Participants from multiple LWIAs may use their ITAs on a specific provider’s program, so it is possible that 
their choices may be represented in the responses of more than one LWIA. 


Additionally, high-volume training programs may be different than lower volume training programs in their use 
of TBL and their characteristics. For example, certain types of programs may have the ability to draw and serve 
a large number of participants. 
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As part of the U.S. Department of Labor, Employment and Training Administration’s Technology-Based 
Learning Grants, the Orange County, California Workforce Investment Board, with Coastline 
Community College, created a blended training program to improve communication skills of nurses 
for whom English was not the native language. In-person instruction in English language 


fundamentals and pronunciation were supplemented with practice in an English-as-a-Second-Language 
Virtual Hospital. Learners created avatars in Second Life and practiced communicating with their 
instructor, other learners, other medical staff members, patients, and patients’ families in simulated 
scenarios developed by subject matter expert in nursing (Dunham et al. 2011; U.S. DOL 2008a). 


6.1 Local Criteria for TBL-based Training Programs 


Prior to examining the extent and nature of technology-based skills building used for training, it is 
important to briefly examine the local context and factors that may potentially shape it. As noted earlier, 
WIA places no restrictions on the specific use of technology for training services (see Chapter 3). 
However, like state policymakers, LWIBs can establish criteria related to technology use in eligible 
training programs. 


Survey respondents were asked whether their LWIB specified any additional restrictions or requirements 
related to the use of technology for training above and beyond federal and state guidelines. Overall, more 
than one in five respondents (21 percent) reported that their LWIB had established such criteria. These 
included: 


e Participants must have a cumulative GPA of 3.0 or higher, based on at least 24 credit hours and be 
in good standing with the college or university. Students already enrolled must be in the WIA 
program for at least one quarter or semester before being approved to take a distance learning 
course. 


e Participants must justify the need for using a TBL-based training program as opposed to an in- 
person program. 


e ATBL program must be affiliated with a traditional provider (e.g., an online course offered by a 
university in place of a classroom course). 


e Online courses must be provided by an educational institution or have a process to provide 
documentation of attendance, participation, and completion. 


These types of criteria may establish standards for the use of technology for WIA-funded training 
services. While they may restrict initial access to training programs using technology, they may also 
promote technology-based skills building in certain situations (e.g., for more advanced participants, from 
certain types of providers). These standards may then contribute to the patterns described below for 
technology-based skills building use in the high-volume programs. 
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6.2 Prevalence of TBL in Training Programs 


Survey respondents were asked to use the following definitions regarding how training was provided, 
with distance learning (in both distance and blended programs) serving as a proxy for the use of 
technology-based skills building.*’ 


e Distance programs are those delivered online or by electronic linkages (e.g., videoconference) only; 
e Blended programs are those that offered courses or sessions both by distance and in person; and 


e In-person programs are those that relied on face-to-face instruction only (i.e., traditional, classroom- 
based programs); no course or session was held online or by distance. 


To determine whether technology-based skills building was used in a particular training program, LWIA 
respondents were encouraged to consult local delivery staff, program data, and training program 
providers. 


Exhibit 6.1 shows that technology-based skills building was used, to varying degrees, in the form of 
blended approaches to training. Among these programs, one-quarter delivered their services using 
technology with 24 percent using a blended delivery model and 1 percent implementing exclusively 
distance programs. Again, it is important to remember that the characteristics of the five highest volume 
training programs may not represent all eligible training programs. 


Exhibit 6.1: Delivery mode of high-volume training programs 


Distance, 1% _—_—__ | 


Blended, 24% 


In-person, 
75% 


Notes: n=2,014 


‘| Pre-testing indicated that this was the highest level of detail about technology-based skills building that could 


reasonably be expected from survey respondents. 
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High-volume training programs represent the occupational training programs cited by respondents as serving the 
highest number of their AJCs’ participants using ITAs. Respondents could provide information on up to five 
occupational training programs. 

Additional detail provided in Exhibit A.6.1 in Appendix A 

Source: TBL local survey 


To better understand how and where technology-based skills building was being implemented, 
respondents described the provider type, industry sector, and the credentials offered by high-volume 
training programs. 


6.2.1 Prevalence of TBL in training for different provider types 


WIA specifies a wide range of organizations eligible to receive WIA funds for providing training 
activities. These may include, among others, public and private universities and colleges; community 
colleges; state or local education agencies; registered apprenticeship programs; and proprietary 
institutions. Different types of organizations may have varying levels of capacity or motivation to 
provide training using technology-based skills building approaches. For instance, survey research 
conducted for the Sloan Consortium suggests that post-secondary institutions embraced the use of online 
education: in fall 2011, 32 percent of students in degree-granting, post-secondary institutions took at least 
one distance (online) course.” That level of investment was largely mirrored in the data collected by the 
current study. 


As shown on Exhibit 6.2, respondents reported that the largest proportion of the high-volume training 
programs (54 percent) was offered by public, post-secondary educational institutions. While it is likely 
that these providers were primarily community colleges, they may also include public universities and 
any other institution(s) eligible to receive funds under Title IV of the Higher Education Act. Among 
these programs, 30 percent reported using some technology-based component that blends distance or 
online instruction with in-person instruction. Another third (29 percent) of the high-volume programs 
were operated by private, for-profit, or proprietary schools.*’ Among programs at private, for-profit, or 
proprietary schools, 12 percent utilized technology-based skills building in a blended format. 


42 Allen and Seaman, 2013. 


‘8 This group does not include private, nonprofit schools. 
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Exhibit 6.2: Delivery mode of high-volume training programs, by provider type 


Public, post-secondary (54% of programs) 


Distance, Private, for-profit, or 
> 1% proprietary (29%) 
0% _— 
Blended, 
30% 
87% 
In-person, 
0, 
70% SAO coaie 
80% 


Other providers (15%) 


Notes: n=1,019 

High-volume training programs represent the occupational training programs cited by respondents as serving the 
highest number of their AJCs’ participants using ITAs. Respondents could provide information on up to five 
occupational training programs. 

Size of each pie chart is approximately proportional to the percentage of programs for each provider type. 
Additional detail provided in Exhibit A.6.2 in Appendix A 

Source: TBL local survey 


6.2.2 Prevalence of TBL in training for different industry sectors 


Through a range of initiatives, ETA has emphasized training and employment in a number of industry 
sectors as “high-growth, high-demand, and economically vital sectors of the American economy.” It 
includes 14 sectors (e.g., biotechnology, energy, financial services) that were either projected to add 
substantial numbers of new jobs to the economy, projected to affect the growth of other industries, or 
were existing or emerging businesses being transformed by technology and innovation requiring new sets 
of skills for workers. Given the projected demand for workers in these sectors or the sectors’ dependence 
on technology, it is useful to understand whether and how technology-based skills building was used to 
train workers within local workforce systems. 


Many of the high-volume programs reported by respondents were concentrated in a small number of 
industry sectors. Moreover, while the proportion of exclusively distance learning programs was 
negligible across all industries, there were differences in the use of blended delivery by industry sector 
(Exhibit 6.3). For example, one in five health care programs (20 percent), which represent just under half 


“4 E.g., U.S. DOL, 2009. 
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(49 percent) of high-volume programs, were delivered using blended approaches. Transportation 
programs (15 percent of high-volume programs) were less likely to use technology-based skills building: 
5 percent of those programs utilize blended delivery. In contrast, programs in information technology 
(IT), which represent nearly one-tenth (9 percent) of high-volume programs, were more likely to use 
technology-based skills building. Of these IT programs, 47 percent were delivered as blended programs 
and another 3 percent used exclusively distance learning. 


Exhibit 6.3: Delivery mode of high-volume training programs, by industry sector 


Heath care (49% of programs) Transportation (15%) 
Distance, ———— an 
Hn 0% _— 5% 
Blended, 
20% 95% 
Information 
0 
In-person, Oe ag 


79% 
50% 


Notes: n=1,865 

High-volume training programs represent the occupational training programs cited by respondents as serving the 
highest number of their AJCs’ participants using ITAs. Respondents could provide information on up to five 
occupational training programs. 

Size of each pie chart is approximately proportional to the percentage of programs for each provider type. 
Additional detail provided in Exhibit A.6.3 in Appendix A 

Source: TBL local survey 


6.2.3. Prevalence of TBL in training for different credential types 


Finally, it is important to know the types of credentials associated with technology-based skills building 
use in high-volume, WIA-funded training programs. This potentially provides additional insight into the 
types of programs that may be well suited to the adoption of technology-based skills building. Overall, 
the survey data indicated that most high-volume programs were likely to lead to a credential upon their 
successful completion (Exhibit 6.4). Nearly half of these programs (49 percent) were reported as likely to 
lead to occupational skills licenses (e.g., Licensed Nurse Practitioner, Commercial Driver’s License) and 
another 40 percent reported to lead to an occupational skills certificate or credential (e.g., Microsoft 
Certified Systems Engineer, Occupational Safety and Health Administration Construction Safety 
Certification). Further, programs leading to an occupation-specific credential were reported to be less 
likely to incorporate technology-based skills building. Respondents reported that programs that 
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culminated in an Associate’s degree were significantly more likely to use technology-based skills 
building than were other programs. Of high-volume programs leading to an Associate’s degree, 47 
percent used blended delivery. In contrast, 26 percent of those high-volume programs leading to an 
occupational skills credential or certificate and 18 percent of those leading to an occupational skills 
license used blended delivery. 


Exhibit 6.4: Delivery mode of high-volume training programs, by credential received upon 
successful completion 


Occupational skills license Occupational skills 
(49% of programs) credential or certificate 
Distance, ———— (40%) 
1% 1% __— 
Blended, 
18% 
In- 73% 
person, 
82% 
—___ 1% 


Associate’s 
degree (22%) 


53% 


Notes: n=1,936 

High-volume training programs represent the occupational training programs cited by respondents as serving the 
highest number of their AJCs’ participants using ITAs. Respondents could provide information on up to five 
occupational training programs. 

Size of each pie chart is approximately proportional to the percentage of programs for each provider type. 
Additional detail provided in Exhibit A.6.4 in Appendix A 

Source: TBL local survey 


6.3 Summary 


Drawing on the local-level perspective of LWIA respondents sheds light on the use of TBL in the training 
programs under WIA. Among the programs with the largest number of AJC participants in each LWIA, 
approximately one in four used technology-based skills building. Independent of provider, industry, and 
credential, the type of technology-based skills building overwhelmingly favored by the high-volume 
training programs was blended delivery, which combines distance and in-person learning. 
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However, there was considerable variation in the frequency of blended programs among the high-volume 
training programs. Programs within more traditional academic settings (i.e., those in post-secondary 
institutions and those leading to two-year degrees) were more likely to use blended learning approaches. 
Additionally, the data suggest that high growth industries more reliant on technology (e.g., information 
technology, health care) were also more likely to rely on technology-based skills building. 


6. Technology-Based Learning for WIA-Funded Training Services pg. 52 


Exploring the Role and Adoption of Technology-Based Training and Employment Services 


7. Technology-Based Learning for WIA Title Il Services 


As authorized in the WIA, Title II supports the provision of adult education services, family literacy 
services, and English literacy programs. Title II services are designed to ensure that participants have the 
educational foundation needed to pursue employment opportunities that can lead to economic self- 
sufficiency. 


Title II is administered by OVAE at the U.S. Department of Education. It involves a state grants program 
in which the state agency with authority for WIA Title II (e.g., a state department of education) receives 
federal grant monies to fund local service providers to implement adult education services. Local Title II 
programs include local education agencies, community colleges, community-based organizations, 
libraries, volunteer literacy programs, and other non-profit organizations. AJCs can refer participants to 
adult education providers for instruction in Adult Basic Education (ABE), General Educational 
Development (GED) preparation, and English as a Second Language (ESL). This chapter examines the 
extent to which TBL, particularly technology-based skills building, was incorporated into the services 
delivered by the Title II-funded providers to whom AJCs referred participants. 


LWIA respondents were asked to provide information on (up to) five Title II service providers to which 
their AJCs referred the most participants during PY 2011. Because the survey showed that AJCs made 
referrals to a median of only three Title II providers, the data are likely reflective of the universe of Title 
II providers that served LWIA respondents’ AJCs and their participants. Throughout this chapter, it is 
important to note that the unit of analysis is the service provider or organization. A given provider may 
offer a range of Title II services, including ABE, GED test preparation, and ESL programs. 


Ludlow Adult Learning Center (MA) incorporates blended and electronic-only delivery to instruct 
WIA, Title II English as a Second Language (ESL) learners and prepare them for transitions to college 
and careers. Technology literacy is embedded into English-language instruction, as learners are required 
to submit, provide feedback on, and revise assignments electronically. This both exposes learners to 
vocabulary and computer-based reading and writing and teaches job or college-readiness skills. In 
class instruction is further supplemented with computer-aided blended instruction. This course, held in a 


computer lab, allows learners to further their learning at their own pace with support from an in-person 
instructor. Learners may use Burlington English, which presents language then uses speech-recognition 
software to analyze learners’ oral language and provide instruction on correct pronunciation and 
comprehensibility, and U.S.A. Learns, a scaffolded, video-based course offered free to learners by the 
U.S. Department of Education (Alamprese et al., forthcoming). 


This chapter documents the prevalence of technology-based skills building among the high-volume Title 
II providers and emphasizes their organizational type and their service mix (i.e., the primary purpose of 
referrals). To determine whether technology-based skills building was used by particular Title II 
providers, LWIA respondents were encouraged to consult with local delivery staff, LWIB data, and Title 
II providers. The survey questions used the definitions of delivery mode described in Chapter 6, with 
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“distance learning” (either distance or blended services) serving as a proxy for the use of technology- 
based skills building.” 


7.1. Prevalence of TBL in High-Volume WIA Title II Programs 


As was the case with training programs under WIA Title I, technology-based skills building approaches 
were used by a minority of the high-volume WIA Title II providers reported in the survey. These 
approaches were predominantly blended, and very few Title II providers offer exclusively distance 
learning services. Among the high-volume Title II providers, nearly one-third delivered their services 
using TBL. That is, 31 percent used a blended delivery model—combining both in-person and distance 
instruction—and another 1 percent were exclusively distance providers (Exhibit 7.1). 


Exhibit 7.1: Delivery mode of high-volume Title II providers 


Distance, - —-— 
1% 


Blended, 31% 


In-person, 
69% 


Notes: n=1,079 

High-volume Title II providers represent the Title Il-funded providers cited by respondents as receiving referrals from 
their AJCs. Respondents could provide information on up to five Title Il-funded providers. 

Additional detail provided in Exhibit A.7.1 in Appendix A 

Source: TBL local survey 


To better understand how the characteristics of the high-volume Title II providers relate to their use of 
technology-based skills building, LWIA respondents were asked to describe each of these programs in 
terms of provider type and primary purpose for the referral. 


4 Pilot testing indicated that this was the highest level of detail about technology-based skills building used by 


Title II providers that could reasonably be expected from survey respondents. 
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7.1.1 Prevalence of TBL for Title Il services for different provider types 


As with training providers (Chapter 6), different types of organizations may differ in their ability and 
interest in using technology-based skills building approaches. As shown in Exhibit 7.2, local education 
agencies (e.g., school districts) represent the most prominent type of provider among the high-volume 
Title II providers cited in the surveys. Among local education agency providers (63 percent of high- 
volume providers), 35 percent used blended approaches for Title II services with the remainder being 
offered exclusively in person. Other types of providers were less likely to use technology-based skills 
building in the provision of Title II services. For instance, 26 percent of institutions of higher education 
(13 percent of providers), such as community colleges, used technology-based skills building to deliver 
Title II services. As with training service providers under WIA, traditional educational institutions were 
most likely to have the infrastructure and knowledge to provide technology-based skills building 
opportunities for Title II learners. 


Exhibit 7.2: Delivery mode of high-volume Title II providers, by provider type 


Local education agency 


(63% of providers) 
Distance, ._ ————__ Institute of 
>1% higher education 
(13%) 
——__ 0% 
74% 

—— 0% 

In-person, > 

65% 


80% 


Community-based 
organization (9%) 


Notes: n=1,068 

High-volume Title II providers represent the Title Il-funded providers cited by respondents as receiving referrals from 
their AJCs. Respondents could provide information on up to five Title Il-funded providers. 

Additional detail provided in Exhibit A.7.2 in Appendix A 

Source: TBL local survey 


7.1.2. Prevalence of TBL for Title Il services for different services 


Respondents were asked to list the primary purpose of the referrals—ABE, GED, and/or ESL—for each 
high-volume Title II provider. The referral purpose is interpreted as a proxy for the content or types of 
adult education services offered by the providers. Referrals for a given provider may be made for a 
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number of purposes, so the services described here are not mutually exclusive. As shown in Exhibit 7.3, 
there was little variation in the use of technology-based skills building by provider across types of 
services. Among programs that received referrals for any of these services, approximately one-third 
utilized technology-based skills building, particularly blended approaches. 


Exhibit 7.3: Delivery mode of high-volume WIA Title II providers, by purpose of referrals 


Adult Basic Education General Educational Development 
(ABE; 69% of providers) (GED; 60%) 
Distance, _—_—___~ >1%_____—— 
1% 
Blended, 
32% 
64% 
In- 
person, 
67% 


—__ 0% 


English as a Second 
Language (ESL; 28%) 


65% 


Notes: n=1,246-1,251 

High-volume Title II providers represent the Title Il-funded providers cited by respondents as receiving referrals from 
their AJCs. Respondents could provide information on up to five Title II-funded providers. 

Percentages across purpose do not sum to 100 since respondents could select multiple responses. 

Additional detail provided in Exhibit A.7.3 in Appendix A 

Source: TBL local survey 


7.2 Summary 


In sum, approximately one-third of Title II service providers reported by survey respondents incorporated 
technology-based skills building into their service provision. Largely, these providers were local 
education agencies, 35 percent of whom offer blended services. Among the types of Title II services 
provided (ABE, GED, or ESL courses), there was little variation in how much service providers utilize 
technology-based skills building. 
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8. Factors Affecting Successful Implementation of Technology-Based 


Learning 


The integration of TBL into the workforce system is an evolving process that is affected by various 
factors ranging from resource availability to customer needs. To help identify and prioritize these issues, 
both SWA and LWIA representatives were asked to rate the importance of a number of factors in terms of 
whether they pose potential barriers to the adoption and use of TBL.*° The factors were drawn from the 
research literature on TBL and customized to each respondent population. Both surveys asked 
respondents to rate the importance of factors related to the perceived instructional effectiveness of TBL, 
resources or costs of using TBL, and stakeholder acceptance of TBL. Additionally, the LWIA 
respondents were asked to rate the importance of several additional issues that reflect the preferences and 
level of technological readiness of their AJCs’ participant population. Finally, respondents were asked to 
further describe the major factors underlying their current level of TBL implementation. 


8.1 Factors Affecting State-Level Implementation of TBL 


Exhibit 8.1 summarizes SWAs’ rating of potential barriers to the implementation of TBL. Responses 
from SWA respondents suggest acceptance, particularly among state policymakers, as well as training 
providers. Over half of SWA respondents (56 and 55 percent, respectively) said that these groups posed 
no or little barrier to implementing TBL. However, only one-third of respondents (34 percent) indicated 
that employers’ acceptance of TBL posed little or no barrier. Few (no more than 10 percent) SWA 
respondents considered stakeholder acceptance at any level—state, training programs, employers, or the 
workforce system—to be a major barrier to implementation. 


“6 All ratings were based on a five-point Likert scale ranging from “no barrier” to “significant barrier.” 
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Exhibit 8.1: Reported factors affecting state implementation of TBL 


SIGNIFICANT BARRIER NOT A BARRIER 
Stakeholder acceptance of TBL 
State policymakers 1%” 
10% Training programs 32%” 
7% Employers HSME 19% 
The workforce 
: a 26% 
6% investment system 
Technological limitations of TBL 
22% & Usselle e MISUSE 13% 
compatibility 
20% GY Trainingrequirements MBBI1% 


Instructional effectiveness of TBL 
22% SEAM Limitations ofinstruction BRIS%N 


18% M18%l Integrity of tests BM 24% — 


Costs or resources for TBL 


20% 8% Assessing quality 12% 
33% 2% implementingtechnology MBDI% 
Shortage of approved wm 18% 
18% "a%i providers 


21% Mi7% Developing courses @ 25% — 
100% 80% 60% 40% 20% 0% 20% 40% 60% 80% 100% 


Notes: n-24-37 

Responses with less than 5 percent are not labeled. 

Darker shade corresponds to the strongest response (significant barrier/no barrier); lighter color corresponds to a 
more moderate response. 

This graphic omits the middle category of moderate barrier. 

Additional detail provided in Exhibit A.8.1 in Appendix A 

Source: TBL state survey 


Against this receptive backdrop, SWA respondents did express concerns regarding both costs and the 
instructional effectiveness of TBL. Their responses reflected concerns about actual expenditures on TBL 
and the organizational cost of transitioning to the use of TBL. Approximately 44 percent of SWA 
respondents reported that the cost or difficulty of implementing the necessary technology for TBL courses 
was a large or significant barrier. The cost or difficulty of developing TBL courses was also seen as a 
large or significant barrier by 38 percent of SWA respondents. These findings were reflected in 
respondents’ qualitative answers. The most common factor cited by state respondents related to the 
required resources or costs of TBL. For instance: 


e Technology Coordinators are challenged by costs associated with supporting not only several 
platforms for customers, but by supporting multiple versions of the same platforms. 


e Some local workforce investment areas have been reluctant to enroll participants in online training 
programs due to the higher cost often involved in these programs and challenges to monitoring 
participation in those offerings. 
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One-third (36 percent) of SWA respondents noted that the perceived limitations of online instruction were 
large or significant barriers to broader use of TBL. A similar proportion felt that concerns about the 
integrity of tests or assessments were also a large or significant barrier. In the open ended follow-up 
question, respondents identified additional factors related to instructional effectiveness that might not 
have been addressed in their ratings. Specifically, the respondents pointed to participants’ levels of 
technological literacy and access to the required technology (particularly Internet connections) as major 
factors limiting their states’ current level of TBL. For example: 


e Some prevalent barriers include the participant's comfort with technology and the availability of 
computers for all participants. 


e The use and expansion of TBL is directly impacted by the availability of broadband technology in 
various locations across the state, the digital divide and a level of digital illiteracy among a specific 
job seeking customer base, and the rapidly changing face of technology. 


Finally, in their open-ended responses, a few SWA respondents also pointed to efforts to mitigate these 
challenges related to technological literacy. For example: 


e [While] the AJCs are all equipped with computer labs, not all participants are comfortable 
completing whole programs of study in public areas. Workforce staff members often offer 
participants computer literacy courses to increase individual proficiency prior to enrolling 
participants in full courses of study. This is beginning to alleviate the fear and worry about working 
in a public space. 


8.2 Factors Affecting Local-Level Implementation of TBL 


LWIA respondents also were asked to rate a similar, but expanded, set of factors and perceived barriers to 
the local adoption and use of TBL. Consistent with the LWIA respondents’ responsibilities, these factors 
focused on issues relevant to their AJCs’ client base, including participants’ access to equipment and their 
perceived technological readiness to pursue TBL. In addition, the survey also asked about issues 
addressed by LWIA respondents’ state-level counterparts, including the perceived effectiveness of TBL 
instruction, the cost of implementation, and general stakeholder acceptance. At the local level, as at the 
state level, responses varied greatly both within and across the categories of responses. 
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Exhibit 8.2: Reported factors affecting local implementation of TBL 


SIGNIFICANT BARRIER NOT A BARRIER 
Stakeholder acceptance of TBL 
12% State policymakers 20% 
16% Training programs 20% 
The workforce investment oa 
ae system ns 
15% Employers 


Technological limitations of TBL 
13% Technological compatability [504 


19% Training requirements 10% 


Technology access or readiness 


Access for those with special 


11% sunita PEYS 
1 — a 
32% Level — j= 
Instructional effectiveness of TBL 
16% [REY Integrity of tests 16% 


25% Limiations of instruction 3a 16% 
18% Preferences for in-person 
30% Low level of effort —12% 
31% Course drop out — 13% 


Costs or resources for TBL 


18% Enrolling students 10% 
21% Assessing quality SSNS 
21% Shortage of approved 10% 
providers 
22% 27% Meeting TBL standards Ex 12% | 
25% Implementing technology 
23% Developing courses (410% 
100% 80% 60% 40% 20% 0% 0% 20% 40% 60% 80% 100% 


Percent of LWIA respondents 


Notes: n=218-385 

Responses with less than 5 percent are not labeled. 

Darker shade corresponds to the strongest response (significant barrier/no barrier); lighter color corresponds to a 
more moderate response. 

This graphic omits the middle category of moderate barrier. 

Additional detail provided in Exhibit A.8.2 in Appendix A 

Source: TBL local survey 
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LWIA respondents’ perceptions generally aligned with those of the SWA respondents. LWIA 
respondents reported that the workforce system operated in an environment that was receptive to TBL 
adoption and use (Exhibit 8.2). Acceptance among key stakeholders was consistently rated as being little 
or no barrier to TBL implementation. For example, fewer than one in four LWIA respondents considered 
acceptance by state policymakers, degree or credential-granting programs, or the workforce system to be 
a large or significant barrier to TBL implementation (18, 24, and 16 percent, respectively). Like their 
state counterparts, LWIA respondents expressed some uncertainty over employer acceptance; however, 
this was not at a level that suggests it was a great impediment to moving forward (29 percent considered 
this a large or significant barrier). In their open-ended responses LWIA respondents pointed to the 
importance of stakeholder acceptance as a facilitator of TBL. For example: 


e Inorder for TBL to be successful in the LWIA, both job-seekers and employers need buy-in to 
TBL. 

e [Our state’s TBL is] initiated and supported by resources developed with WIA [such as the] 10 
percent set-aside funds and national discretionary grants; resources residing on [our state] 
knowledge management portal and ETPL web site; and initiatives undertaken at the LWIB level. 


Among LWIA respondents, three broad factors were reported as barriers that warrant further exploration 
and attention. One concern was the technological readiness of participants. Specifically, nearly two- 
thirds (63 percent) of respondents cited participants’ level of technological literacy as a large or 
significant barrier to TBL implementation. This perception was echoed in LWIA respondents’ responses 
to the open-ended question. For instance: 


e The major factor our AJCs face when increasing the usage of TBL is the overall lack of computing 
skills by a large percentage of our clients. 


e Much of our job seeker population does not have the technology skills required to fully participate in 
TBL. 


e The AJC conducts weekly basic computer skills training to address the needs of youth and adult 
jobseekers. While both groups may text or have Facebook accounts, they do not have the necessary 
skills to conduct an online job search, receive electronic notices of job openings, or complete 
resumes. 


This is a particularly noteworthy issue that was also raised by state administrators. Clearly, this issue was 
of sufficiently widespread concern to have generated awareness outside of practitioner and operational 
circles. A second, and perhaps related, concern had to do with perceptions regarding the effectiveness of 
instruction in a technology-based setting. As shown in Exhibit 8.2, 42 percent of respondents viewed the 
limitations of TBL instruction to be a large or significant barrier. This sentiment was underscored by the 
related perception that technology-based instruction may be less capable of fully engaging learners, 
resulting in lower levels of effort (e.g., social loafing; 51 percent of LWIA respondents) or course dropout 
(52 percent of LWIA respondents). 


The roles of these perceived barriers are further complicated by perceptions about the participants’ skills 
or abilities and their preferences for in-person instruction. This was reported to be a large or significant 
barrier by nearly half of the LWIA respondents (48 percent). It is important to note that these concerns 
were not necessarily seen as inherent limitations of TBL, but rather challenges related to effectively 
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serving their target populations. In describing these concerns about instructional effectiveness, 
respondents noted: 


e Many students need the extra face-to-face instruction to succeed. 


e Weare serving people with significant barriers and the perceived time management skills to be 
disciplined and successfully complete are not adequate...The web opens a wide variety of possibilities 
and dangers. Our area needs to explore more but are a conservative group and shy of the dangers. 


e Weare starting to use more technology as part of our service delivery (computer lab instruction, 
social media, etc.). We have been leery of computer-based instruction—we have used it in the past, 
and it has been difficult to keep students motivated and engaged. 


e The participants in our system seem to still have a higher preference for in-person communication or 
training. There can sometimes be challenges in participants successfully completing TBL 
coursework, as there are diverse levels of comfort in utilizing TBL technology and courses. 


Finally, some of the highest percentages of LWIAs respondents reported significant barriers related to the 
required resources or costs of TBL. In particular, approximately half of the LWIA respondents (between 
45 and 55 percent) cited the costs of developing TBL courses, implementing the necessary technology for 
TBL courses, meeting TBL standards, and assessing the quality of TBL courses to be large or significant 
barriers. This pattern was repeated in the qualitative data. The plurality of respondents cited the required 
resources or costs of TBL, and the most common single answer to the open-ended question was simply 
“cost.” Other respondents elaborated: 


e It is costly to buy and requires frequent updating and maintenance. 


e The major underlying factor is cost. Partners are not willing or able to share in the cost of providing 
the infrastructure for significant TBL within the comprehensive and satellite [American Job Centers]. 


e TBL is not cost-effective at this time. With the current WIA budget cuts, it is impossible to purchase 
the technology and to develop curriculum for TBL. 


e [The] additional costs associated with TBL due to the need for program-related equipment and 
software required to participate in TBL [are a major barrier]. 


These responses from LWIA respondents aligned with similar responses from SWA respondents. 
Together, these data suggests that, despite the long-term potential of TBL for promoting efficiency, the 
developmental and transitional investments needed to incorporate TBL into workforce development were 
a challenge. 


Despite the myriad challenges that local delivery systems faced in implementing TBL, several LWIA 
respondents pointed to proactive efforts—often bolstered by extant resources, external funds or 
partnerships—to incorporate TBL within their LWIAs. For example: 


e [One LWIB] led the [nearby] area in “sharing” [resources] with the contiguous workforce 
boards. Many services were purchased with unlimited users in order to scale the license to ‘pay 
for itself.’ 


e [Major factors underlying our current use of TBL are] collaborative partnerships among state 
and local agencies to leverage resources. 
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e Costs and staff time to convert workshop material is an item that we are addressing locally and 
are looking for no or low cost programs...as well as looking at community partnerships to 
leverage resources for our mutual customers. 


Many LWIA respondents were optimistic about the role of TBL within local workforce development. As 
one respondent reflected: 


The desire to move forward with more TBL is there. However, finding out what's out there and 
determining what is more effective becomes a challenge. A better awareness of new products and 
methods will increase everyone's interest for more TBL. We are looking for innovative ways to build 
more TBL into our programming and enhance our service delivery. 


8.3. Summary 


Stakeholders at both the state and local levels generally agreed on the major factors and challenges 
underlying the successful use of TBL within the workforce system. Both sets of survey respondents 
indicated that the major issues were more operational than attitudinal. Stakeholder acceptance— 
including that by policymakers, program providers, and employers—was seen as little or no barrier to 
TBL implementation. 


Against this broad backdrop of acceptance, respondents identified a number of challenges to be 
addressed. The first and most noteworthy perceived barriers were cost related. While TBL may have the 
potential to promote cost efficiency over time, there were infrastructure, development, and transitional 
costs that need to be addressed before these efficiencies will be realized. The second key barrier cited by 
both state and local respondents was the technological readiness of participants. Despite proactive 
investments in assessments, orientations, and technical assistance, technological readiness of participants 
was a concern and potential barrier to further adoption of TBL. The third perceived barrier to TBL 
adoption and use was the quality and effectiveness of instruction and the related challenge of maintaining 
the necessary level of student engagement. This was raised as a particular concern in effectively serving 
target populations with educational or skills deficits. 
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9. Implications 


The Department is committed to promoting the use of TBL by sponsoring TBL-related projects and 
research, communicating TBL’s potential, and sharing effective practices. This study, part of ETA’s TBL 
Initiative, provides a snapshot of state and local workforce system stakeholders’ perceptions about the use 
of TBL and current activities within the workforce system. When appropriate, the analyses in this report 
distinguished between technology-based service and technology-based skills building. The responses 
from both state and LWIA representatives suggest that a number of conditions exist which may support 
the continued integration of TBL into the workforce system. 


State-level policymakers, as represented through the responses of SWA administrators, were aware of 
TBL and its potential to expand the reach, flexibility, and efficiency of workforce services. Nearly half of 
the states responding to the survey (44 percent) had made institutional commitments to TBL in the form 
of policies, legislation, and/or investments that promote TBL. Seventy three percent of state respondents 
acknowledged the complementary roles that TBL plays in supporting other strategic objectives for the 
workforce system. Additionally, TBL had achieved a level of acceptance among key stakeholders, 
including the network of state policymakers, training providers and, to a slightly lesser extent, businesses. 


While the pace of adoption varied considerably, LWIBs also made commitments and investments. This 
was occurring both within the AJCs and through their training and education providers. The AJCs have 
direct control over the technology component of core and intensive services. Three-quarters of the LWIA 
respondents reported that their AJCs offered blended services that combine both electronic and in-person 
support, primarily for technology-based services (71 percent). Further, one in five (19 percent) of the 
respondents reported that their LWIBs contracted with a virtual AJC as an online option for accessing 
services. In comparison, training providers eligible for WIA funding appeared to be moving at a more 
deliberate pace in integrating technology-based skills building into their occupational training programs. 
Approximately one-quarter (24 percent) of high-volume occupational training programs offered some 
form of blended training. This use of technology-based skills building varied somewhat by type of 
provider and the nature of the occupational training. A similar pattern was found in the provision of adult 
education services through Title II, where approximately one-third (31 percent) of high-volume Title II 
providers offered some form of blended adult education services. 


Exhibit 9.1: Factors related to current and future TBL in the workforce system 


Facilitators of TBL adoption and use Areas for additional work 

Development of infrastructure through leveraging Continued identification of cost-sharing 

existing resources opportunities 

Integration of personal support Improved individual engagement and retention 

Use of a digital “on-ramp” Improved understanding of technological literacy 

Active promotion and marketing of TBL Improved insight into the dynamics of TBL 
promotion 

Commitment to technical assistance Improved understanding of what makes for 


effective technical assistance 
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This research helped to identify several issues that should be further explored as ETA refines the focus of 
its TBL Initiative going forward. Overall, the research points to several inter-related factors related to 
current and future adoption and use of TBL in the workforce system. These factors are summarized in 
Exhibit 9.1 and discussed in detail below. 


Facilitators to TBL Adoption and Use 


This research provides insight into several design factors that appear to be instrumental in promoting the 
adoption and use of TBL. The following themes emerged across the survey responses and the qualitative 


Development of infrastructure through new investments and leveraging existing resources. 
Survey data indicated concerns about the costs of implementing effective TBL. For example, the 
necessary infrastructure for TBL (e.g., up-to-date computers, Internet connections) was not 
universally available. The states and LWIBs that made commitments to TBL appeared to 
recognize that infrastructure investments are necessary in order to accommodate those on the 
other side of the “digital divide.” These populations may include both those in rural or outlying 
areas as well as those facing other barriers to accessing technology. Many AJCs invested in 
computer equipment and Internet access for participants. These resources can be deployed 
flexibly to support participants’ engagement with other technology-based activities, including 
TBL. Respondents also reported concerns related to assessing the effectiveness of TBL training 
programs. AJCs’ relationships with eligible training providers can help to reduce the cost of 
assessing course quality when these approved providers begin to offer technology-based skills 
building courses. 


Integration of personal support. In many instances, the transition to technology-based services 
and technology-based skills building appears to be aided by a simultaneous commitment to 
personal (i.e., real-time) support. In considering the use of TBL at the local level, it is important 
to make the distinction between provision and engagement. Survey responses suggest that a 
number of stakeholders were concerned with issues of participant engagement. That is, many 
perceive that workforce participants lack the focus, motivation, knowledge, and/or skills base 
needed to successfully pursue TBL without an appropriate level of personal support. This 
support can come through many design components including blended instruction, proactive case 
management, or resources for tutoring or mentoring. 


Use of a digital “on-ramp”. Adoption of a more TBL-based service strategy must be seen as a 
deliberately paced effort that is attentive to participants’ levels of technological literacy. Gaining 
the necessary level of functionality and comfort may be best achieved through some type of 
flexible “on-ramp” that ensures that participants have the tools and confidence needed to succeed 
in a technologically-based environment. Key resources that can be flexibly applied include, for 
instance, some combination of assessments, orientations, technology skills building, or technical 
support and assistance. 


Active promotion and marketing of TBL. While reliance on technology has become the norm 
in countless dimensions of daily life, such integration nonetheless represents an ongoing and 
protracted sea change in the world of training and education. The results from this research help 
to highlight the challenge of bridging the gap between the availability of TBL and the use of 
TBL. LWIA respondents appear to recognize this challenge and have been proactive about 
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9.2 


efforts to expand awareness, consideration, and use. This includes efforts to promote awareness 
and interest among users as well as setting realistic expectations of providers (i.e., expectations of 
skills attainment regardless of training mode). 


Commitment to technical assistance. Approximately two-thirds of the LWIAs reported that 
providing some form of technical assistance to support users of TBL. This strategy is likely 
intended to encompass two intertwined objectives: 1) to provide the support needed for AJC 
participants to reap maximum benefit from WIA services and 2) to create and communicate a 
supportive learning environment that will attract and retain participants in the future. 


Areas for Additional Work 


While the data illuminated facilitators of TBL, at the same time the responses from SWAs and LWIA 
respondents shed light on several related areas for additional work moving forward. Areas that will 
provide guidance to those looking to refine ETA’s TBL Initiative going forward include: 


Continued identification of cost-sharing opportunities. Despite the potential of TBL to 
promote cost efficiency and increase access to services, state and LWIA respondents confirm that 
cost-related factors are seen as barriers to increased investments in TBL. These concerns clearly 
reflect the budget-constrained environment in which they operate and the primarily low-income 
populations served by the workforce system. It is important that the TBL Initiative play a lead 
role in exploring institutional options to share developmental costs and defray operating costs. 
This includes the expansion of efforts to, for instance, underwrite developmental costs that 
benefit the system; share content across states, LWIAs, and providers; share technological 
platforms; and harness large-scale purchasing power. ETA continues to make these types of 
investments through, for example, the TAACCCT grants, which encourage grantees, in 
collaboration with community colleges, “to develop online training programs that build on 
current advances in science and technology and are scalable to large numbers of [Trade 
Adjustment Assistance] eligible workers and other adults.””” 


Improved individual engagement and retention. A major challenge facing successful TBL use 
is to fully engage AJC participants. LWIA respondents did not appear to have full confidence in 
the quality of TBL-based instruction. They expressed this sentiment by acknowledging concerns 
that TBL may result in problematic levels of social loafing or dropout. It is important to examine 
this issue further to determine whether the bigger challenge is the quality of synchronous 
technology-based instruction (e.g., webinars, virtual classrooms) or the inherent capacity of 
asynchronous modes (e.g., full self-study courses, podcasts) to fully engage students at a 
meaningful level. In any event, addressing this challenge requires working with training and 
education providers to further convey expectations or standards around TBL, such as those 
related to course design and pilot testing; training or certification specific to using technology for 
services or skills building; and the availability of student support options (e.g., learning 
communities). 


Improved understanding of technological literacy. Both state and LWIA respondents 
expressed similar sentiments that suggest that participants’ levels of technological literacy (also 


4” U.S. DOL, 2013a, p 6. 
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referred to as “digital literacy”) is a barrier to more robust adoption of TBL. Given this pervasive 
sentiment, it is important to gain additional insight into exactly what constitutes an appropriate 
level of technological literacy and where the current skills gaps are. Further, it is important to 
examine this issue in the context of current and emerging technologies, as well as any 
implications for making use of technology-based services or technology-based skills building 
activities. Finally, the use of technology for workforce development —whether increasing access 
or building knowledge through technology based-services or through technology-based skills 
building—may affect participants’ levels of technological literacy and prepare them for additional 
engagement in TBL. Future work may explore strategies for increasing technological literacy, 
including through the use of TBL itself. 


e Improved insight into the dynamics of TBL promotion. While promotional efforts are well 
underway, much remains to be understood about the dynamics of this process. For instance, what 
it takes to establish an enticing and technology-friendly service-delivery environment, how to use 
existing technology to promote newer technology, and how to best break down lingering concerns 
among employers about the quality of credentials earned or skills gained through TBL. It is 
important that these issues be explored in the context of DOL’s well-established electronic 
network (e.g., Workforce3One, communities of practice associated with particular grants) and its 
social media footprint. 


e Improved understanding of what makes for effective technical assistance. Despite the 
prominence of the commitment to technical assistance, it is not clear how LWIAs view and 
prioritize these objectives. In addition, it would be valuable to further examine which technical 
assistance strategies appear to be most effective in promoting these aims. An inquiry into the 
focus and effectiveness of technical assistance efforts can also shed light on potential strategies 
for bridging the technology literacy gap discussed above. 


Finally, the workforce system has a long established history of providing individualized and hands-on 
support to those looking to enhance their employability and career opportunities. The overriding 
emphasis on blended options underscores the importance of implementing TBL at a pace and in such a 
way that maintains the essential character of the workforce system. In this context it is important that 
ETA’s TBL Initiative remains aware of and sensitive to establishing realistic and balanced expectations 
around the pace of TBL adoption. Integral to this is ETA’s challenge to develop an appropriately 
balanced message to policymakers, planners, practitioners, and providers. While this research suggests 
that state and local stakeholders embraced the value and potential of TBL, it is also essential that ETA 
establish a message that avoids any implication that more is better or sooner is better. Rather, ETA can 
continue to offer both the insight and support needed for state decision-makers and LWIBs to make 
informed decisions about where and when to invest in TBL. 
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